[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:18 dist-test-slave-tc2g krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:18 dist-test-slave-tc2g krb5kdc[399](info): set up 2 sockets
Jun 20 14:31:18 dist-test-slave-tc2g krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:31:19.442205 392 mini_kdc.cc:121] Time spent starting KDC: real 2.514s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 14:31:19.591701 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.149s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 14:31:19.700425 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.109s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:31:20.008529 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.308s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/test-user.keytab.
W20260620 14:31:20.688668 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.680s user 0.000s sys 0.001s
Jun 20 14:31:20 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:20.730152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:20.809725 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:20.809949 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:20.809984 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:20.811131 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:20.811198 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:20.811223 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:20.811251 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:20.811275 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:20.812736 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:20.813007 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:20.813269 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:20.818552 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:20.818591 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:20.818567 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:20.818724 412 server_base.cc:1034] running on GCE node
I20260620 14:31:20.819083 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:20.819398 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:20.820569 412 hybrid_clock.cc:648] HybridClock initialized: now 1781965880820541 us; error 39 us; skew 500 ppm
Jun 20 14:31:20 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:20.823079 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:20.823565 412 webserver.cc:460] Webserver started at http://127.0.98.62:46761/ using document root <none> and password file <none>
I20260620 14:31:20.823797 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:20.823865 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:20.824015 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:20.825073 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "147a895b8e5e48c480832d2e36393a44"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "1cc0a9557d14cb74e8e175e3a8a732a4"
server_key_iv: "54e4686d1b734de95079760103217cc7"
server_key_version: "encryptionkey@0"
I20260620 14:31:20.825987 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "147a895b8e5e48c480832d2e36393a44"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "1cc0a9557d14cb74e8e175e3a8a732a4"
server_key_iv: "54e4686d1b734de95079760103217cc7"
server_key_version: "encryptionkey@0"
I20260620 14:31:20.827474 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:20.828243 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:20.828414 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:20.828513 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "147a895b8e5e48c480832d2e36393a44"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "1cc0a9557d14cb74e8e175e3a8a732a4"
server_key_iv: "54e4686d1b734de95079760103217cc7"
server_key_version: "encryptionkey@0"
I20260620 14:31:20.828604 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:20.839564 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:20.841838 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:20.841995 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:20.845890 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45883
I20260620 14:31:20.845919 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45883 every 8 connection(s)
I20260620 14:31:20.846246 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:20.847041 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:20.849217 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44: Bootstrap starting.
I20260620 14:31:20.849833 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:20.850059 480 log.cc:824] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:20.850636 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44: No bootstrap required, opened a new log
I20260620 14:31:20.852123 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } }
I20260620 14:31:20.852252 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:20.852274 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147a895b8e5e48c480832d2e36393a44, State: Initialized, Role: FOLLOWER
I20260620 14:31:20.852407 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } }
I20260620 14:31:20.852491 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:20.852514 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:20.852541 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:20.852990 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } }
I20260620 14:31:20.853070 480 leader_election.cc:302] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147a895b8e5e48c480832d2e36393a44; no voters:
I20260620 14:31:20.853210 480 leader_election.cc:288] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:20.853250 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:20.853380 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [term 1 LEADER]: Becoming Leader. State: Replica: 147a895b8e5e48c480832d2e36393a44, State: Running, Role: LEADER
I20260620 14:31:20.853482 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:20.853574 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } }
I20260620 14:31:20.854405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 412
I20260620 14:31:20.854449 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147a895b8e5e48c480832d2e36393a44. Latest consensus state: current_term: 1 leader_uuid: "147a895b8e5e48c480832d2e36393a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } } }
I20260620 14:31:20.854516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:20.854522 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:20.854679 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147a895b8e5e48c480832d2e36393a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147a895b8e5e48c480832d2e36393a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45883 } } }
I20260620 14:31:20.854738 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:20.854987 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:20.855325 392 external_mini_cluster.cc:1428] Setting key 36ea837f573ee15ec2cb5fc9828d188e
I20260620 14:31:20.855566 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:20.857911 492 catalog_manager.cc:1347] Generated new cluster ID: 21fc3452a02b491cb5f6b644eec728bb
I20260620 14:31:20.857963 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:20 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:20.871173 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:20.871860 492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:20.873687 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:20.856168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.0.98.62:45883)
0620 14:31:20.856337 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:20.856347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:20.856669 (+ 322us) server_negotiation.cc:407] Connection header received
0620 14:31:20.859057 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:20.859064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:20.859116 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:20.859249 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:20.861849 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.863265 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.864806 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.865389 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.865521 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:20.869751 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:20.869755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:20.869755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:20.869767 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:20.871795 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:20.872291 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:20.872294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:20.872294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:20.872379 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:20.872631 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:20.872632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:20.872632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:20.872835 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:20.872985 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:20.873271 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:31:20.873438 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 14:31:20.879830 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147a895b8e5e48c480832d2e36393a44: Generated new TSK 0
I20260620 14:31:20.880007 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:20.908264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:20.988535 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:20.988732 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:20.988759 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:20.989936 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:20.989976 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:20.990003 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:20.991446 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:20.991681 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:20.991925 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:20.996783 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:20.996802 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:20.996802 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:20.997053 506 server_base.cc:1034] running on GCE node
I20260620 14:31:20.997200 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:20.997380 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:20.998512 506 hybrid_clock.cc:648] HybridClock initialized: now 1781965880998499 us; error 30 us; skew 500 ppm
Jun 20 14:31:20 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:21.000962 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:21.001390 506 webserver.cc:460] Webserver started at http://127.0.98.1:34695/ using document root <none> and password file <none>
I20260620 14:31:21.001632 506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:21.001677 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:21.001832 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:21.002892 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "520c76142a1646fa8573d73804040880"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "cf527e9a22a8a4f06e54e45e35642e75"
server_key_iv: "80d50b7e46473b3835ab136176237e42"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.003247 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "520c76142a1646fa8573d73804040880"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "cf527e9a22a8a4f06e54e45e35642e75"
server_key_iv: "80d50b7e46473b3835ab136176237e42"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.004522 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:21.005203 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.005405 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:21.005551 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "520c76142a1646fa8573d73804040880"
format_stamp: "Formatted at 2026-06-20 14:31:20 on dist-test-slave-tc2g"
server_key: "cf527e9a22a8a4f06e54e45e35642e75"
server_key_iv: "80d50b7e46473b3835ab136176237e42"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.005614 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:21.017485 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:21.019248 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:21.019402 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:21.019618 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:21.019934 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:21.019968 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.020009 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:21.020033 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.026042 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36357
I20260620 14:31:21.026090 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36357 every 8 connection(s)
I20260620 14:31:21.026386 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:21.032680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 506
I20260620 14:31:21.032783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:21.032934 392 external_mini_cluster.cc:1428] Setting key e57854b008828eda447ece741f4e045f
I20260620 14:31:21.035559 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.027036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43885 (local address 127.0.98.62:45883)
0620 14:31:21.027107 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.027109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.027311 (+ 202us) server_negotiation.cc:407] Connection header received
0620 14:31:21.027758 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.027760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.027771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.027822 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.028710 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.029203 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.030250 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.030459 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.030518 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.032568 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:21.032570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:21.032570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:21.032577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:21.034114 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.034592 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.034592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.034593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.034639 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.034909 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.034909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.034910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.035050 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:21.035121 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:21.035351 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:31:21.035444 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:21.035569 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.027026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45883 (local address 127.0.98.1:43885)
0620 14:31:21.027192 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:21.027227 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:21.027677 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:21.027895 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:21.027898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:21.028457 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:21.028636 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.028641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.029277 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.029278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.030157 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.030160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.030536 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.030536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.030631 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.031067 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.031075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.032480 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:21.034195 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.034201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.034203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.034515 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.034700 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.034701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.034701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.034836 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.035120 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:21.035124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:21.035229 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:31:21.035316 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:31:21.035424 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 14:31:21.036028 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45883
I20260620 14:31:21.036118 635 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:21.036305 635 heartbeater.cc:511] Master 127.0.98.62:45883 requested a full tablet report, sending...
I20260620 14:31:21.036787 444 ts_manager.cc:194] Registered new tserver with Master: 520c76142a1646fa8573d73804040880 (127.0.98.1:36357)
I20260620 14:31:21.037495 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:21.056228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:21.135553 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:21.135784 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:21.135824 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:21.137012 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:21.137075 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:21.137121 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:21.138655 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:21.138923 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:21.139146 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:21.144127 640 server_base.cc:1034] running on GCE node
W20260620 14:31:21.144107 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:21.144099 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:21.144100 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:21.144528 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:21.144718 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:21.145875 640 hybrid_clock.cc:648] HybridClock initialized: now 1781965881145864 us; error 29 us; skew 500 ppm
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:21.148219 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:21.148622 640 webserver.cc:460] Webserver started at http://127.0.98.2:42201/ using document root <none> and password file <none>
I20260620 14:31:21.148873 640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:21.148940 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:21.149084 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:21.150180 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "365ca66ce5824bfd8355581fd91aa70b"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "9af705e8c51155cc5bcf385fa9e7462f"
server_key_iv: "5c9786566167d855b6adc0870641d7f2"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.150537 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "365ca66ce5824bfd8355581fd91aa70b"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "9af705e8c51155cc5bcf385fa9e7462f"
server_key_iv: "5c9786566167d855b6adc0870641d7f2"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.151772 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:21.152407 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.152561 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:21.152652 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "365ca66ce5824bfd8355581fd91aa70b"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "9af705e8c51155cc5bcf385fa9e7462f"
server_key_iv: "5c9786566167d855b6adc0870641d7f2"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.152726 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:21.169872 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:21.171725 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:21.171877 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:21.172096 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:21.172396 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:21.172466 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.172506 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:21.172534 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.178916 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42229
I20260620 14:31:21.178987 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42229 every 8 connection(s)
I20260620 14:31:21.179303 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:21.180454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 640
I20260620 14:31:21.180569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:21.180755 392 external_mini_cluster.cc:1428] Setting key b0dd2fc2ef3b7fe671e5127583cd6c05
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:31:21.189625 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.180008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44179 (local address 127.0.98.62:45883)
0620 14:31:21.180050 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.180052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.180228 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:31:21.180836 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.180838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.180852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.180936 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.181961 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.182745 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.184319 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.184496 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.184544 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.186833 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:21.186834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:21.186834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:21.186842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:21.188247 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.188662 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.188664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.188664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.188724 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.188972 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.188973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.188973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.189084 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:21.189162 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:21.189454 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:31:21.189535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:21.189808 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.179964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45883 (local address 127.0.98.2:44179)
0620 14:31:21.180110 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:21.180160 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:21.180714 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:21.181011 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:21.181016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:21.181673 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:21.181860 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.181865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.182844 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.182845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.184235 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.184238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.184574 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.184575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.184721 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.185216 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.185227 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.186741 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:21.188367 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.188373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.188375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.188585 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.188797 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.188798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.188798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.188919 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.189153 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:21.189159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:21.189302 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:31:21.189403 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:31:21.189579 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260620 14:31:21.190367 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45883
I20260620 14:31:21.190446 769 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:21.190577 769 heartbeater.cc:511] Master 127.0.98.62:45883 requested a full tablet report, sending...
I20260620 14:31:21.190936 444 ts_manager.cc:194] Registered new tserver with Master: 365ca66ce5824bfd8355581fd91aa70b (127.0.98.2:42229)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:31:21.191390 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:21.204165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:21.286702 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:21.286906 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:21.286929 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:21.288103 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:21.288144 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:21.288170 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:21.289620 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:21.289793 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:21.289958 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:21.295123 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:21.295655 774 server_base.cc:1034] running on GCE node
W20260620 14:31:21.295642 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:21.295135 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:21.296000 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:21.296236 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:21.297394 774 hybrid_clock.cc:648] HybridClock initialized: now 1781965881297385 us; error 39 us; skew 500 ppm
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:21.299826 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:21.300273 774 webserver.cc:460] Webserver started at http://127.0.98.3:37179/ using document root <none> and password file <none>
I20260620 14:31:21.300495 774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:21.300556 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:21.300694 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:21.301815 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "602043c7393242f1a05dfafa8fe3598a"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "4a91d17ab42376d2b726866f26def88d"
server_key_iv: "0c1f1814d9a35fd1799a3ccb768bb2ac"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.302163 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "602043c7393242f1a05dfafa8fe3598a"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "4a91d17ab42376d2b726866f26def88d"
server_key_iv: "0c1f1814d9a35fd1799a3ccb768bb2ac"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.303457 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:21.304138 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.304311 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:21.304402 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "602043c7393242f1a05dfafa8fe3598a"
format_stamp: "Formatted at 2026-06-20 14:31:21 on dist-test-slave-tc2g"
server_key: "4a91d17ab42376d2b726866f26def88d"
server_key_iv: "0c1f1814d9a35fd1799a3ccb768bb2ac"
server_key_version: "encryptionkey@0"
I20260620 14:31:21.304481 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:21.325981 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:21.327771 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:21.327931 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:21.328171 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:21.328531 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:21.328583 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.328625 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:21.328650 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:21.335542 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46341
I20260620 14:31:21.335640 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46341 every 8 connection(s)
I20260620 14:31:21.335938 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:21.338596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 774
I20260620 14:31:21.338675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:21.338842 392 external_mini_cluster.cc:1428] Setting key 60bbfb509e095cf89d0cac450cf4d2a7
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:21.345009 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.336655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42871 (local address 127.0.98.62:45883)
0620 14:31:21.336713 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.336715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.336916 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:31:21.337456 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.337457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.337468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.337510 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.338447 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.339088 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.340074 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.340275 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.340355 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.342497 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:21.342499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:21.342499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:21.342506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:21.343942 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.344281 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.344281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.344282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.344329 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.344522 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.344523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.344523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.344626 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:21.344674 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:21.344887 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:31:21.344949 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:31:21.345232 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.336629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45883 (local address 127.0.98.3:42871)
0620 14:31:21.336788 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:21.336836 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:21.337336 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:21.337590 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:21.337594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:21.338150 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:21.338340 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.338345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.339173 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.339174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.339970 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.339972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.340365 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.340366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.340485 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.340917 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.340926 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.342406 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:21.344012 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.344017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.344018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.344217 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.344383 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:21.344383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:21.344383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:21.344472 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:21.344677 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:21.344681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:21.344795 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:31:21.344873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:31:21.344982 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 14:31:21.345762 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45883
I20260620 14:31:21.345858 903 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:21.346030 903 heartbeater.cc:511] Master 127.0.98.62:45883 requested a full tablet report, sending...
I20260620 14:31:21.346398 444 ts_manager.cc:194] Registered new tserver with Master: 602043c7393242f1a05dfafa8fe3598a (127.0.98.3:46341)
I20260620 14:31:21.346755 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42871
I20260620 14:31:21.350642 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:21.373314 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.365588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.0.98.1:36357)
0620 14:31:21.365756 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.365762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.365772 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:21.365797 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.365798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.365813 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.365877 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.366450 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.367138 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.368328 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.368632 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.368687 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.370812 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:21.370819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:21.370819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:21.370825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:21.372407 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.372663 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.372665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.372665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.372709 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.372844 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.372845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.372845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.372964 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:21.373066 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:21.373123 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:31:21.373180 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
W20260620 14:31:21.373708 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:21.394472 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.387873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.0.98.1:36357)
0620 14:31:21.387934 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.387936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.388002 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:31:21.388104 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.388105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.388122 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.388187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.388712 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.389183 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.389964 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.390166 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.390268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.392353 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:21.392355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:21.392356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:21.392363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:21.393664 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.393947 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.393947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.393948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.393993 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:21.394132 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:21.394132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:21.394132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:21.394206 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:21.394286 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:21.394358 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:31:21.394413 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:31:21.395010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 506
I20260620 14:31:21.400098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 640
I20260620 14:31:21.405493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 774
I20260620 14:31:21.409704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 412
2026-06-20T14:31:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4507 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[917](info): set up 2 sockets
Jun 20 14:31:21 dist-test-slave-tc2g krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 14:31:22.444996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:22.557883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:22.637640 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:22.637878 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:22.637916 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:22.639081 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:22.639159 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:22.639185 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:22.639212 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:22.639243 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:22.640723 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:22.640992 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:22.641225 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:22.646078 930 server_base.cc:1034] running on GCE node
W20260620 14:31:22.646057 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:22.646062 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:22.646051 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:22.646557 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:22.646857 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:22.648020 930 hybrid_clock.cc:648] HybridClock initialized: now 1781965882648012 us; error 42 us; skew 500 ppm
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:22.650381 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:22.650794 930 webserver.cc:460] Webserver started at http://127.0.98.62:45031/ using document root <none> and password file <none>
I20260620 14:31:22.651024 930 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:22.651090 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:22.651237 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:22.652282 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "4837367bd1044a59bd86067037bd8c56"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "25d88d4d299ff73d8f31df794f09062a"
server_key_iv: "c0ab5e4025005e40f56c19fde5a0723b"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.652748 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "4837367bd1044a59bd86067037bd8c56"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "25d88d4d299ff73d8f31df794f09062a"
server_key_iv: "c0ab5e4025005e40f56c19fde5a0723b"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.654145 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:22.654872 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:22.655069 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:22.655174 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "4837367bd1044a59bd86067037bd8c56"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "25d88d4d299ff73d8f31df794f09062a"
server_key_iv: "c0ab5e4025005e40f56c19fde5a0723b"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.655246 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:22.683837 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:22.685730 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:22.685904 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:22.690265 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35725
I20260620 14:31:22.690302 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35725 every 8 connection(s)
I20260620 14:31:22.690686 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:22.691519 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:22.692508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 930
I20260620 14:31:22.692605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:22.692759 392 external_mini_cluster.cc:1428] Setting key 0ff2a76703b5dd17a51bf55365232c00
I20260620 14:31:22.694229 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56: Bootstrap starting.
I20260620 14:31:22.694975 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:22.695283 998 log.cc:824] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:22.696013 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56: No bootstrap required, opened a new log
I20260620 14:31:22.698446 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } }
I20260620 14:31:22.698622 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:22.698663 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837367bd1044a59bd86067037bd8c56, State: Initialized, Role: FOLLOWER
I20260620 14:31:22.698822 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } }
I20260620 14:31:22.698915 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:22.698951 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:22.699007 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:22.699645 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } }
I20260620 14:31:22.699741 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4837367bd1044a59bd86067037bd8c56; no voters:
I20260620 14:31:22.699882 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:22.699977 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:22.700093 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [term 1 LEADER]: Becoming Leader. State: Replica: 4837367bd1044a59bd86067037bd8c56, State: Running, Role: LEADER
I20260620 14:31:22.700137 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:22.700865 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } }
I20260620 14:31:22.701238 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4837367bd1044a59bd86067037bd8c56. Latest consensus state: current_term: 1 leader_uuid: "4837367bd1044a59bd86067037bd8c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } } }
I20260620 14:31:22.701329 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:22.701627 1017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:22.701797 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4837367bd1044a59bd86067037bd8c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4837367bd1044a59bd86067037bd8c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35725 } } }
I20260620 14:31:22.701889 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:22.702222 1017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:22.704634 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.693270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.0.98.62:35725)
0620 14:31:22.693459 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:22.693469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:22.693501 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:31:22.694026 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:22.694033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:22.694081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:22.694227 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:22.694988 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.696163 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.697190 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.697647 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.697735 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.700633 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:22.700637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:22.700638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:22.700650 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:22.702975 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:22.703388 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:22.703391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:22.703392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:22.703490 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:22.703709 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:22.703710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:22.703710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:22.704005 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:22.704117 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:22.704355 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:31:22.704459 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260620 14:31:22.704957 1017 catalog_manager.cc:1347] Generated new cluster ID: d13bd44747524049be955d302d420a7c
I20260620 14:31:22.705010 1017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:22.723663 1017 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:22.724393 1017 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:22.728355 1017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4837367bd1044a59bd86067037bd8c56: Generated new TSK 0
I20260620 14:31:22.728524 1017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:22.754415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:22.834975 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:22.835163 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:22.835186 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:22.836249 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:22.836287 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:22.836313 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:22.837642 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:22.837815 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:22.837982 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:22.842829 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:22.842849 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:22.842926 1024 server_base.cc:1034] running on GCE node
W20260620 14:31:22.842839 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:22.843322 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:22.843519 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:22.844655 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781965882844644 us; error 27 us; skew 500 ppm
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:22.847097 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:22.847509 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38583/ using document root <none> and password file <none>
I20260620 14:31:22.847738 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:22.847805 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:22.847947 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:22.848989 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "cd63352629772b54e3c76a6b3444293c"
server_key_iv: "2c0c35b8919507cbfb172a11fb0daa66"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.849345 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "cd63352629772b54e3c76a6b3444293c"
server_key_iv: "2c0c35b8919507cbfb172a11fb0daa66"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.850615 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:22.851325 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:22.851512 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:22.851613 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "cd63352629772b54e3c76a6b3444293c"
server_key_iv: "2c0c35b8919507cbfb172a11fb0daa66"
server_key_version: "encryptionkey@0"
I20260620 14:31:22.851689 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:22.866819 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:22.868618 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:22.868780 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:22.869010 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:22.869320 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:22.869375 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:22.869436 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:22.869485 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:22.875025 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36741
I20260620 14:31:22.875101 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36741 every 8 connection(s)
I20260620 14:31:22.875448 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:22.878876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1024
I20260620 14:31:22.878952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:22.879118 392 external_mini_cluster.cc:1428] Setting key e7491f0c035d017ec9ed40411e6e0316
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:22.885347 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.876174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33627 (local address 127.0.98.62:35725)
0620 14:31:22.876233 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:22.876235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:22.876448 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:31:22.876969 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:22.876970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:22.876982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:22.877045 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:22.878051 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.878562 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.879812 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.880032 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.880116 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.882601 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:22.882603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:22.882603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:22.882609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:22.883952 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:22.884440 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:22.884440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:22.884441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:22.884487 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:22.884773 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:22.884773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:22.884773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:22.884864 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:22.884910 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:22.885228 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:31:22.885286 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:31:22.885633 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.876139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35725 (local address 127.0.98.1:33627)
0620 14:31:22.876333 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:22.876368 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:22.876866 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:22.877120 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:22.877124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:22.877757 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:22.877950 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.877956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.878661 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.878662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.879725 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.879727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.880125 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.880126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.880254 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.880771 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:22.880780 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:22.882503 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:22.884032 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:22.884039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:22.884041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:22.884364 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:22.884558 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:22.884560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:22.884560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:22.884704 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:22.884923 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:22.884928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:22.885091 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:31:22.885208 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:31:22.885378 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260620 14:31:22.886160 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35725
I20260620 14:31:22.886255 1153 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:22.886427 1153 heartbeater.cc:511] Master 127.0.98.62:35725 requested a full tablet report, sending...
I20260620 14:31:22.886955 962 ts_manager.cc:194] Registered new tserver with Master: 7adde3bb2c4e4b3abf9fd59f11a313a1 (127.0.98.1:36741)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:31:22.887766 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:22.903672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:22.982596 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:22.982777 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:22.982800 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:22.983901 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:22.983933 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:22.983959 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:22.985253 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:22.985474 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:22.985663 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:22.990424 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:22.990504 1158 server_base.cc:1034] running on GCE node
W20260620 14:31:22.990430 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:22.990430 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:22.990903 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:22.991142 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:22.992280 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781965882992268 us; error 28 us; skew 500 ppm
Jun 20 14:31:22 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:22.999168 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:22.999540 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38999/ using document root <none> and password file <none>
I20260620 14:31:22.999724 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:22.999763 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:22.999848 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:23.000913 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "0cbc6f54880346939b2aab6b43a68696"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "814793866bba1dbe3fdb615399052dbd"
server_key_iv: "145062ec652293e10c9270fc6ede7221"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.001269 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0cbc6f54880346939b2aab6b43a68696"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "814793866bba1dbe3fdb615399052dbd"
server_key_iv: "145062ec652293e10c9270fc6ede7221"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.002526 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:23.003165 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.003355 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:23.003440 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "0cbc6f54880346939b2aab6b43a68696"
format_stamp: "Formatted at 2026-06-20 14:31:22 on dist-test-slave-tc2g"
server_key: "814793866bba1dbe3fdb615399052dbd"
server_key_iv: "145062ec652293e10c9270fc6ede7221"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.003510 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:23.014361 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:23.016708 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:23.016872 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:23.017097 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:23.017400 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:23.017472 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.017516 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:23.017544 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.023387 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39131
I20260620 14:31:23.023424 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39131 every 8 connection(s)
I20260620 14:31:23.023753 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:23.028008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1158
I20260620 14:31:23.028098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:23.028303 392 external_mini_cluster.cc:1428] Setting key ab6db9ac4190379415f14b79b32f0797
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:23.033238 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.024478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51227 (local address 127.0.98.62:35725)
0620 14:31:23.024520 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.024522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.024704 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:31:23.025166 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.025167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.025177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.025246 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.026191 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.026664 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.027576 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.027758 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.027812 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.030308 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.030310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.030310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.030319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.031791 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.032301 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.032302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.032302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.032350 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.032616 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.032617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.032617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.032710 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.032757 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.033097 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.033159 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:23.033593 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.024457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35725 (local address 127.0.98.2:51227)
0620 14:31:23.024587 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.024620 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.025088 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.025303 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.025306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.025923 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:23.026119 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.026124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.026736 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.026737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.027505 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.027507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.027853 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.027854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.027994 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.028575 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:23.028583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:23.030167 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:23.031873 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:23.031879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:23.031881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:23.032197 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:23.032415 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:23.032415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:23.032416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:23.032552 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:23.032773 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:23.032779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:23.032945 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.033082 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.033287 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260620 14:31:23.034113 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35725
I20260620 14:31:23.034207 1287 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:23.034425 1287 heartbeater.cc:511] Master 127.0.98.62:35725 requested a full tablet report, sending...
I20260620 14:31:23.034801 962 ts_manager.cc:194] Registered new tserver with Master: 0cbc6f54880346939b2aab6b43a68696 (127.0.98.2:39131)
I20260620 14:31:23.035168 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51227
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:23.051697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:23.132577 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:23.132800 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:23.132840 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:23.134038 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:23.134099 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:23.134145 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:23.135583 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:23.135828 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:23.136050 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:23.140740 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:23.140787 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:23.140796 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:23.140875 1292 server_base.cc:1034] running on GCE node
I20260620 14:31:23.141076 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:23.141285 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:23.142423 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781965883142407 us; error 32 us; skew 500 ppm
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:23.144860 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:23.145329 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41195/ using document root <none> and password file <none>
I20260620 14:31:23.145568 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:23.145634 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:23.145778 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:23.146797 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "d665c5a53bcc4db0aa8e4352371e6adc"
format_stamp: "Formatted at 2026-06-20 14:31:23 on dist-test-slave-tc2g"
server_key: "75e38672e0e9114a5d442f935c08fd82"
server_key_iv: "7ffdcef5c0db03c0e81ae366aee24338"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.147143 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d665c5a53bcc4db0aa8e4352371e6adc"
format_stamp: "Formatted at 2026-06-20 14:31:23 on dist-test-slave-tc2g"
server_key: "75e38672e0e9114a5d442f935c08fd82"
server_key_iv: "7ffdcef5c0db03c0e81ae366aee24338"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.148398 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:23.149125 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.149295 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:23.149389 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "d665c5a53bcc4db0aa8e4352371e6adc"
format_stamp: "Formatted at 2026-06-20 14:31:23 on dist-test-slave-tc2g"
server_key: "75e38672e0e9114a5d442f935c08fd82"
server_key_iv: "7ffdcef5c0db03c0e81ae366aee24338"
server_key_version: "encryptionkey@0"
I20260620 14:31:23.149480 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:23.167196 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:23.168941 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:23.169102 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:23.169341 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:23.169683 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:23.169742 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.169782 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:23.169811 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:23.175439 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38475
I20260620 14:31:23.175482 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38475 every 8 connection(s)
I20260620 14:31:23.175776 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:23.176173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1292
I20260620 14:31:23.176290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:23.176472 392 external_mini_cluster.cc:1428] Setting key 5fc9ac58cac33b60776e05b97622d7a8
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:23.184525 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.176504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55759 (local address 127.0.98.62:35725)
0620 14:31:23.176557 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.176559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.176955 (+ 396us) server_negotiation.cc:407] Connection header received
0620 14:31:23.177559 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.177560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.177573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.177615 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.178556 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.179056 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.179987 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.180167 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.180220 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.182354 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.182355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.182355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.182361 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.183483 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.183813 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.183814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.183814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.183859 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.184052 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.184052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.184052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.184140 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.184186 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.184401 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.184465 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:31:23.184715 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.176498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35725 (local address 127.0.98.3:55759)
0620 14:31:23.176795 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.176843 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.177456 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.177695 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.177698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.178245 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:23.178445 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.178450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.179136 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.179137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.179913 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.179915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.180234 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.180235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.180335 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.180876 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:23.180885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:23.182267 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:23.183553 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:23.183558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:23.183559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:23.183752 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:23.183912 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:23.183913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:23.183913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:23.184003 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:23.184189 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:23.184192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:23.184299 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.184376 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.184492 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":85,"threads_started":1}
I20260620 14:31:23.185200 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35725
I20260620 14:31:23.185277 1421 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:23.185489 1421 heartbeater.cc:511] Master 127.0.98.62:35725 requested a full tablet report, sending...
I20260620 14:31:23.185876 962 ts_manager.cc:194] Registered new tserver with Master: d665c5a53bcc4db0aa8e4352371e6adc (127.0.98.3:38475)
I20260620 14:31:23.186244 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55759
I20260620 14:31:23.188364 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:23.195928 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.189902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.62:35725)
0620 14:31:23.189977 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.189979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.190099 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:31:23.190172 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.190173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.190183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.190238 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.190758 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.191208 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.191936 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.192143 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.192220 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.193360 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.193361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.193362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.193368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.194788 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.195146 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.195147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.195147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.195204 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.195407 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.195407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.195408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.195557 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.195653 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.195756 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.195831 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:23.198125 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:23.199589 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:23.206203 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.202155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.0.1:36146)
0620 14:31:23.202236 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.202247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.202308 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.202870 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.202870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.202872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:23.202914 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.202916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.204148 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.204149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.205058 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.205060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.205892 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.205981 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.205996 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.206076 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.206125 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":49,"threads_started":1}
I20260620 14:31:23.206223 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.202329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.0.98.3:38475)
0620 14:31:23.202521 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.202525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.202534 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:23.202568 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.202570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.202585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.202657 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:23.202986 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.204044 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205151 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.205817 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205890 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.205943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.206098 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.206137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20260620 14:31:23.206619 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.201914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36741 (local address 127.0.0.1:40310)
0620 14:31:23.202065 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.202097 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.202189 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.202482 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.202486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.202489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:23.202558 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.202560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.204006 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.204007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.205361 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.205362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206359 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.206419 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.206426 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.206487 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.206538 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260620 14:31:23.206653 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.202036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.98.1:36741)
0620 14:31:23.202208 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.202214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.202226 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:23.202279 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.202281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.202297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.202372 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:23.202734 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.203908 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205466 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206178 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206258 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.206318 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.206510 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.206566 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 14:31:23.206782 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.202504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.0.98.2:39131)
0620 14:31:23.202653 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.202658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.202854 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:31:23.202900 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.202902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.202916 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.202993 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:23.203108 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.203763 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205352 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206270 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206340 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.206395 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.206661 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.206705 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260620 14:31:23.206801 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.202393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39131 (local address 127.0.0.1:43146)
0620 14:31:23.202748 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.202759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.202814 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.203003 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.203004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.203007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:23.203052 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.203053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.203869 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.203870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.205242 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.205244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206426 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.206517 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.206522 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.206577 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.206620 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":31,"thread_start_us":48,"threads_started":1}
I20260620 14:31:23.207340 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c2d00def028646ef86041ecd904c3305 (DEFAULT_TABLE table=test-table [id=af8c9fc1de354180a265f1fab9635e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:23.207355 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c2d00def028646ef86041ecd904c3305 (DEFAULT_TABLE table=test-table [id=af8c9fc1de354180a265f1fab9635e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:23.207432 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c2d00def028646ef86041ecd904c3305 (DEFAULT_TABLE table=test-table [id=af8c9fc1de354180a265f1fab9635e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:23.207623 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d00def028646ef86041ecd904c3305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:23.207682 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d00def028646ef86041ecd904c3305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:23.207695 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d00def028646ef86041ecd904c3305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:23.210186 1435 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Bootstrap starting.
I20260620 14:31:23.210762 1435 tablet_bootstrap.cc:654] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:23.211242 1436 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Bootstrap starting.
I20260620 14:31:23.211251 1435 log.cc:824] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:23.211781 1436 tablet_bootstrap.cc:654] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:23.212289 1435 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: No bootstrap required, opened a new log
I20260620 14:31:23.212368 1435 ts_tablet_manager.cc:1397] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:23.212354 1436 log.cc:824] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:23.212661 1437 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Bootstrap starting.
I20260620 14:31:23.213020 1436 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: No bootstrap required, opened a new log
I20260620 14:31:23.213104 1436 ts_tablet_manager.cc:1397] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:23.213265 1437 tablet_bootstrap.cc:654] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:23.213804 1437 log.cc:824] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:23.214859 1436 raft_consensus.cc:348] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.215041 1436 raft_consensus.cc:374] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:23.215076 1437 tablet_bootstrap.cc:492] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: No bootstrap required, opened a new log
I20260620 14:31:23.214936 1435 raft_consensus.cc:348] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.215080 1436 raft_consensus.cc:729] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d665c5a53bcc4db0aa8e4352371e6adc, State: Initialized, Role: FOLLOWER
I20260620 14:31:23.215113 1435 raft_consensus.cc:374] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:23.215134 1437 ts_tablet_manager.cc:1397] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:31:23.215147 1435 raft_consensus.cc:729] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7adde3bb2c4e4b3abf9fd59f11a313a1, State: Initialized, Role: FOLLOWER
I20260620 14:31:23.215197 1436 consensus_queue.cc:260] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.215272 1435 consensus_queue.cc:260] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.215498 1436 ts_tablet_manager.cc:1428] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:31:23.215526 1435 ts_tablet_manager.cc:1428] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:23.215535 1421 heartbeater.cc:503] Master 127.0.98.62:35725 was elected leader, sending a full tablet report...
I20260620 14:31:23.215646 1153 heartbeater.cc:503] Master 127.0.98.62:35725 was elected leader, sending a full tablet report...
I20260620 14:31:23.217002 1437 raft_consensus.cc:348] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.217125 1437 raft_consensus.cc:374] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:23.217146 1437 raft_consensus.cc:729] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbc6f54880346939b2aab6b43a68696, State: Initialized, Role: FOLLOWER
I20260620 14:31:23.217252 1437 consensus_queue.cc:260] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.217535 1437 ts_tablet_manager.cc:1428] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:23.217557 1287 heartbeater.cc:503] Master 127.0.98.62:35725 was elected leader, sending a full tablet report...
W20260620 14:31:23.274356 1288 tablet.cc:2369] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:23.341256 1442 raft_consensus.cc:482] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:23.341372 1442 raft_consensus.cc:504] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.341856 1442 leader_election.cc:288] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d665c5a53bcc4db0aa8e4352371e6adc (127.0.98.3:38475), 0cbc6f54880346939b2aab6b43a68696 (127.0.98.2:39131)
I20260620 14:31:23.345719 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.342140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48061 (local address 127.0.98.2:39131)
0620 14:31:23.342191 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.342193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.342384 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:31:23.342417 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.342418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.342441 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.342500 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:23.342626 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.343316 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.344719 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.345307 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.345386 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.345397 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.345603 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.345643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:31:23.345710 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.342114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38253 (local address 127.0.98.3:38475)
0620 14:31:23.342169 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.342171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.342210 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:31:23.342241 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.342242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.342260 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.342313 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:23.342505 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.343266 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.344719 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.345343 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.345402 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.345410 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.345603 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.345641 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:31:23.345737 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.342038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.1:38253)
0620 14:31:23.342116 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.342129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.342185 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.342335 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.342336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.342338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:23.342436 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.342437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.343343 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.343344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.344608 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.344612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.345412 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.345413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.345514 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.345521 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.345583 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.345625 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":4}
I20260620 14:31:23.345743 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.342082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39131 (local address 127.0.98.1:48061)
0620 14:31:23.342257 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.342265 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.342346 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.342501 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.342502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.342504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:23.342557 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.342558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.343381 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.343382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.344608 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.344612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.345413 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.345413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.345514 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.345521 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:23.345586 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:23.345625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":61,"thread_start_us":114,"threads_started":1}
I20260620 14:31:23.346118 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d00def028646ef86041ecd904c3305" candidate_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" is_pre_election: true
I20260620 14:31:23.346290 1375 raft_consensus.cc:2454] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7adde3bb2c4e4b3abf9fd59f11a313a1 in term 0.
I20260620 14:31:23.346381 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d00def028646ef86041ecd904c3305" candidate_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbc6f54880346939b2aab6b43a68696" is_pre_election: true
I20260620 14:31:23.346525 1241 raft_consensus.cc:2454] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7adde3bb2c4e4b3abf9fd59f11a313a1 in term 0.
I20260620 14:31:23.346535 1043 leader_election.cc:302] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7adde3bb2c4e4b3abf9fd59f11a313a1, d665c5a53bcc4db0aa8e4352371e6adc; no voters:
I20260620 14:31:23.346649 1442 raft_consensus.cc:2781] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:23.346704 1442 raft_consensus.cc:482] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:23.346721 1442 raft_consensus.cc:3037] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:23.347388 1442 raft_consensus.cc:504] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.347524 1442 leader_election.cc:288] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [CANDIDATE]: Term 1 election: Requested vote from peers d665c5a53bcc4db0aa8e4352371e6adc (127.0.98.3:38475), 0cbc6f54880346939b2aab6b43a68696 (127.0.98.2:39131)
I20260620 14:31:23.347662 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d00def028646ef86041ecd904c3305" candidate_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d665c5a53bcc4db0aa8e4352371e6adc"
I20260620 14:31:23.347723 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d00def028646ef86041ecd904c3305" candidate_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbc6f54880346939b2aab6b43a68696"
I20260620 14:31:23.347743 1375 raft_consensus.cc:3037] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:23.347815 1241 raft_consensus.cc:3037] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:23.348484 1375 raft_consensus.cc:2454] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7adde3bb2c4e4b3abf9fd59f11a313a1 in term 1.
I20260620 14:31:23.348604 1241 raft_consensus.cc:2454] T c2d00def028646ef86041ecd904c3305 P 0cbc6f54880346939b2aab6b43a68696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7adde3bb2c4e4b3abf9fd59f11a313a1 in term 1.
I20260620 14:31:23.348654 1043 leader_election.cc:302] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7adde3bb2c4e4b3abf9fd59f11a313a1, d665c5a53bcc4db0aa8e4352371e6adc; no voters:
I20260620 14:31:23.348745 1442 raft_consensus.cc:2781] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:23.348925 1442 raft_consensus.cc:686] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [term 1 LEADER]: Becoming Leader. State: Replica: 7adde3bb2c4e4b3abf9fd59f11a313a1, State: Running, Role: LEADER
I20260620 14:31:23.349020 1442 consensus_queue.cc:237] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } }
I20260620 14:31:23.349905 962 catalog_manager.cc:5495] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7adde3bb2c4e4b3abf9fd59f11a313a1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7adde3bb2c4e4b3abf9fd59f11a313a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d665c5a53bcc4db0aa8e4352371e6adc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cbc6f54880346939b2aab6b43a68696" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39131 } health_report { overall_health: UNKNOWN } } }
W20260620 14:31:23.376156 1154 tablet.cc:2369] T c2d00def028646ef86041ecd904c3305 P 7adde3bb2c4e4b3abf9fd59f11a313a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:23.395669 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.387584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.98.1:36741)
0620 14:31:23.387689 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.387691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.387792 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:31:23.387884 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.387885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.387900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.387969 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.388559 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.389185 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.390026 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.390264 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.390343 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.392693 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.392704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.392704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.392711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.394453 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.394818 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.394822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.394823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.394902 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.395106 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.395107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.395108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.395258 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.395440 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.395478 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.395552 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:23.418959 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.411583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.0.98.1:36741)
0620 14:31:23.411662 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.411664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.411750 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:31:23.411850 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.411851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.411869 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.411934 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.412501 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.412986 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.413784 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.413985 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.414087 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.416288 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.416289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.416289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.416297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.417762 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.418115 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.418117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.418117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.418179 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.418386 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.418387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.418387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.418485 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.418591 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.418777 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.418856 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:31:23.426291 1422 tablet.cc:2369] T c2d00def028646ef86041ecd904c3305 P d665c5a53bcc4db0aa8e4352371e6adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:23.440394 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.433660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.98.1:36741)
0620 14:31:23.433719 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.433725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.433815 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:31:23.433870 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.433871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.433880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.433938 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.434441 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.434905 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.435603 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.435782 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.435836 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.437905 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:23.437906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:23.437906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:23.437912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:23.439290 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.439594 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.439594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.439595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.439641 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:23.439879 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:23.439880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:23.439883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:23.439975 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:23.440072 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:23.440184 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:31:23.440257 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:31:23.440842 1067 tablet_service.cc:2908] Rejecting scan request for tablet c2d00def028646ef86041ecd904c3305: Uninitialized: clean time has not yet been initialized
I20260620 14:31:23.441644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1024
I20260620 14:31:23.446710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1158
I20260620 14:31:23.451495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1292
I20260620 14:31:23.455705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 930
2026-06-20T14:31:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2049 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[1459](info): set up 2 sockets
Jun 20 14:31:23 dist-test-slave-tc2g krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260620 14:31:24.481016 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:24 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:24.591970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:24.669648 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:24.669890 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:24.669927 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:24.671051 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:24.671121 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:31:24.671144 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:24.671166 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:24.671193 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:24.671214 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:24.672689 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33267
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:24.672971 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:24.673219 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:24.677774 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:24.677809 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:24.677824 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:24.678037 1472 server_base.cc:1034] running on GCE node
I20260620 14:31:24.678270 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:24.678570 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:24.679749 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781965884679718 us; error 41 us; skew 500 ppm
Jun 20 14:31:24 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:24.682240 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:24.682658 1472 webserver.cc:460] Webserver started at http://127.0.98.62:45255/ using document root <none> and password file <none>
I20260620 14:31:24.682875 1472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:24.682945 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:24.683091 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:24.684118 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "cdd040ea2b3ba64ff31b4e84e521f114"
server_key_iv: "c46055a1b1282a9c00c7f79fe80f2f72"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.684491 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "cdd040ea2b3ba64ff31b4e84e521f114"
server_key_iv: "c46055a1b1282a9c00c7f79fe80f2f72"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.685782 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:31:24.686488 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:24.686647 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:24.686748 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "cdd040ea2b3ba64ff31b4e84e521f114"
server_key_iv: "c46055a1b1282a9c00c7f79fe80f2f72"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.686821 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:24.702365 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:24.704584 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:24.704739 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:24.709138 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33267
I20260620 14:31:24.709210 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33267 every 8 connection(s)
I20260620 14:31:24.709602 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:24.710266 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:24.712430 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Bootstrap starting.
I20260620 14:31:24.713021 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:24.713317 1550 log.cc:824] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:24.713969 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: No bootstrap required, opened a new log
I20260620 14:31:24.715931 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } }
I20260620 14:31:24.716063 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:24.716120 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d7e26e4bcd4b18a5b1bd60f9a64526, State: Initialized, Role: FOLLOWER
I20260620 14:31:24.716293 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } }
I20260620 14:31:24.716372 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:24.716421 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:24.716471 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:24.716789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1472
I20260620 14:31:24.716884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:24.717042 392 external_mini_cluster.cc:1428] Setting key e7fa6ac001118c65d93164aecf0bdb3e
I20260620 14:31:24.717072 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } }
I20260620 14:31:24.717185 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d7e26e4bcd4b18a5b1bd60f9a64526; no voters:
I20260620 14:31:24.717353 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:24.717382 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:24.717619 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:24.717813 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [term 1 LEADER]: Becoming Leader. State: Replica: 14d7e26e4bcd4b18a5b1bd60f9a64526, State: Running, Role: LEADER
I20260620 14:31:24.717926 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } }
I20260620 14:31:24.718286 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } } }
I20260620 14:31:24.718367 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:24.718529 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d7e26e4bcd4b18a5b1bd60f9a64526. Latest consensus state: current_term: 1 leader_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d7e26e4bcd4b18a5b1bd60f9a64526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33267 } } }
I20260620 14:31:24.718590 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:24.718942 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:24.719449 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:24.720985 1564 catalog_manager.cc:1347] Generated new cluster ID: f18cb582a2ee4f7eaa934e58e71a510d
I20260620 14:31:24.721036 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:24 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:24.727757 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.717518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.0.98.62:33267)
0620 14:31:24.717766 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.717774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.717819 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:31:24.717937 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.717944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.718006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.718155 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.719203 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.720296 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.721298 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.721861 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.721979 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.724208 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:24.724212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:24.724212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:24.724224 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:24.726250 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.726647 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.726650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.726651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.726739 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.726946 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.726947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.726948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.727182 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:24.727365 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:24.727430 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:31:24.727565 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":53,"threads_started":1}
I20260620 14:31:24.743211 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:24.743903 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:24.752031 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Generated new TSK 0
I20260620 14:31:24.752207 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:24.779477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33267
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:24.859920 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:24.860128 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:24.860152 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:24.861361 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:24.861406 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:24.861476 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:24.861515 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:24.863157 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:24.863426 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:24.863664 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:24.868664 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:24.868691 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:24.868693 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:24.868916 1576 server_base.cc:1034] running on GCE node
I20260620 14:31:24.869143 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:24.869385 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:24.870575 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781965884870564 us; error 30 us; skew 500 ppm
Jun 20 14:31:24 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:24.872931 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:24.873343 1576 webserver.cc:460] Webserver started at http://127.0.98.1:46765/ using document root <none> and password file <none>
I20260620 14:31:24.873589 1576 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:24.873657 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:24.873821 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:24.874915 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "44234692039f4f5b82180b9ba605fd8f"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "4744e63a71257e4a9235f896ab25212f"
server_key_iv: "8206d435fa388b4a3092bad1c84de978"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.875285 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44234692039f4f5b82180b9ba605fd8f"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "4744e63a71257e4a9235f896ab25212f"
server_key_iv: "8206d435fa388b4a3092bad1c84de978"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.876611 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:31:24.877319 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:24.877538 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:24.877637 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "44234692039f4f5b82180b9ba605fd8f"
format_stamp: "Formatted at 2026-06-20 14:31:24 on dist-test-slave-tc2g"
server_key: "4744e63a71257e4a9235f896ab25212f"
server_key_iv: "8206d435fa388b4a3092bad1c84de978"
server_key_version: "encryptionkey@0"
I20260620 14:31:24.877718 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:24.901338 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:24.903163 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:24.903328 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:24.903842 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:24.903908 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:24.903956 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:24.903981 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:24.910671 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37167
I20260620 14:31:24.910749 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37167 every 8 connection(s)
I20260620 14:31:24.911031 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:31:24 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:24.914486 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.904392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58237 (local address 127.0.98.62:33267)
0620 14:31:24.904467 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.904469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.904654 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:31:24.905179 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.905180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.905192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.905257 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.906177 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.907019 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.908642 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.908890 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.908979 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.911641 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:24.911643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:24.911643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:24.911651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:24.913090 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.913536 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.913537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.913538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.913605 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.913868 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.913868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.913869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.913978 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:24.914053 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:24.914315 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:31:24.914399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:24.914556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1576
I20260620 14:31:24.914657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:24.914659 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.904376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.1:58237)
0620 14:31:24.904540 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:24.904578 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:24.905071 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:24.905329 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:24.905340 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:24.905761 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:24.906052 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.906059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.907127 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.907128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.908561 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.908564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.908989 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.908990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.909120 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.909677 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:24.909688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:24.911541 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:24.913191 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:24.913197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:24.913199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:24.913465 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:24.913672 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:24.913673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:24.913673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:24.913788 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:24.914044 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:24.914050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:24.914186 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:31:24.914282 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:31:24.914442 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260620 14:31:24.914839 392 external_mini_cluster.cc:1428] Setting key 6d6ecc105b0f5460b81fd2bc810f0b05
I20260620 14:31:24.915158 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33267
I20260620 14:31:24.915241 1707 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:24.915431 1707 heartbeater.cc:511] Master 127.0.98.62:33267 requested a full tablet report, sending...
I20260620 14:31:24.915946 1504 ts_manager.cc:194] Registered new tserver with Master: 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167)
I20260620 14:31:24.916603 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58237
I20260620 14:31:24.923555 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.916159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55127 (local address 127.0.98.62:33267)
0620 14:31:24.916199 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.916200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.917743 (+ 1543us) server_negotiation.cc:407] Connection header received
0620 14:31:24.917791 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.917792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.917802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.917868 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.918225 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.918812 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.919927 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.920099 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.920152 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.921159 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:24.921160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:24.921160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:24.921166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:24.922520 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.922922 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.922922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.922923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.922969 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:24.923132 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:24.923133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:24.923133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:24.923220 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:24.923267 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:24.923433 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:31:24.923497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:31:24.923672 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.917518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.1:55127)
0620 14:31:24.917658 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:24.917672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:24.917738 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:24.917878 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:24.917878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:24.918096 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:24.918161 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.918163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.918904 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.918906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.919840 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.919843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.920173 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.920174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.920274 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.920653 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:24.920658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:24.921078 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:24.922622 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:24.922624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:24.922624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:24.922849 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:24.923012 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:24.923013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:24.923013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:24.923076 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:24.923285 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:24.923286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:24.923335 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:31:24.923415 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:31:24.923533 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:24.940220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33267
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:25.019212 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:25.019438 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:25.019462 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:25.020586 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:25.020625 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:25.020670 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:25.020714 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:25.022293 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:25.022526 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:25.022773 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:25.027547 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:25.027587 1716 server_base.cc:1034] running on GCE node
W20260620 14:31:25.027557 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:25.027596 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:25.027915 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:25.028144 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:25.029297 1716 hybrid_clock.cc:648] HybridClock initialized: now 1781965885029278 us; error 36 us; skew 500 ppm
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:25.031703 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:25.032109 1716 webserver.cc:460] Webserver started at http://127.0.98.2:38867/ using document root <none> and password file <none>
I20260620 14:31:25.032348 1716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:25.032418 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:25.032557 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:25.033648 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "f6f09eb0be24494da2ecb899210867e2"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "f6bac9784b75dc622b1379ff0a652a1b"
server_key_iv: "17fee7d76a785db7cd12be8d5bf4316e"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.033995 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f6f09eb0be24494da2ecb899210867e2"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "f6bac9784b75dc622b1379ff0a652a1b"
server_key_iv: "17fee7d76a785db7cd12be8d5bf4316e"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.035323 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:25.035993 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.036163 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:25.036257 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "f6f09eb0be24494da2ecb899210867e2"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "f6bac9784b75dc622b1379ff0a652a1b"
server_key_iv: "17fee7d76a785db7cd12be8d5bf4316e"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.036324 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:25.069038 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:25.071476 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:25.071666 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:25.072190 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:25.072254 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.072301 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:25.072324 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:25.079392 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36469
I20260620 14:31:25.079439 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36469 every 8 connection(s)
I20260620 14:31:25.079810 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:25.082466 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.072741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38411 (local address 127.0.98.62:33267)
0620 14:31:25.072790 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.072792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.073017 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:31:25.073588 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.073590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.073604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.073671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.074533 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.075370 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.076981 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.077317 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.077405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.079715 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.079716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.079716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.079724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.081219 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.081654 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.081655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.081655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.081708 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.081901 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.081902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.081902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.081993 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.082038 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.082320 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.082405 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:25.082664 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.072731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.2:38411)
0620 14:31:25.072877 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.072941 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.073414 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.073754 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.073758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.074065 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.074429 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.074435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.075482 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.075484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.076889 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.076892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.077411 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.077412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.077575 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.078135 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.078144 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.079619 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:25.081284 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.081290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.081302 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.081600 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.081766 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.081767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.081767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.081860 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.082040 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:25.082043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:25.082213 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.082288 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.082416 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260620 14:31:25.083187 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33267
I20260620 14:31:25.083267 1847 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:25.083461 1847 heartbeater.cc:511] Master 127.0.98.62:33267 requested a full tablet report, sending...
I20260620 14:31:25.083789 1504 ts_manager.cc:194] Registered new tserver with Master: f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469)
I20260620 14:31:25.084235 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38411
I20260620 14:31:25.085040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1716
I20260620 14:31:25.085131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:25.085307 392 external_mini_cluster.cc:1428] Setting key dc90e352615ff648013953d5204f0031
I20260620 14:31:25.090307 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.084143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.2:60007)
0620 14:31:25.084327 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.084339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.084402 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.084527 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.084528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.084711 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.084757 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.084758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.085610 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.085611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.086388 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.086390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.086701 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.086702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.086769 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.087129 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.087134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.087481 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:25.089230 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.089231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.089231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.089393 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.089655 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.089656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.089656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.089727 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.090074 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:25.090074 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:25.090111 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.090160 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.090234 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260620 14:31:25.090395 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.084280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60007 (local address 127.0.98.62:33267)
0620 14:31:25.084332 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.084334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.084389 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:31:25.084451 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.084452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.084462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.084512 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.084828 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.085521 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.086468 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.086633 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.086681 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.087560 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.087562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.087562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.087570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.089149 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.089520 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.089521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.089522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.089579 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.089797 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.089798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.089799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.090005 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.090085 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.090199 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.090269 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:25.109968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33267
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:25.190531 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:25.190753 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:25.190793 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:25.191943 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:25.192019 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:25.192051 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:25.192088 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:25.193802 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:25.194077 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:25.194317 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:25.199113 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:25.199069 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:25.199051 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:25.199103 1856 server_base.cc:1034] running on GCE node
I20260620 14:31:25.199487 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:25.199738 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:25.200917 1856 hybrid_clock.cc:648] HybridClock initialized: now 1781965885200904 us; error 39 us; skew 500 ppm
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:25.203469 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:25.203900 1856 webserver.cc:460] Webserver started at http://127.0.98.3:37051/ using document root <none> and password file <none>
I20260620 14:31:25.204125 1856 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:25.204191 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:25.204339 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:25.205394 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "a91d1f34c6d64ca38294c090bf04f977"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "d7ad0b2ec572d58411a8569bf626c504"
server_key_iv: "dd4b0311b30eff3dc5d7b674b349f6b7"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.205808 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a91d1f34c6d64ca38294c090bf04f977"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "d7ad0b2ec572d58411a8569bf626c504"
server_key_iv: "dd4b0311b30eff3dc5d7b674b349f6b7"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.207036 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:25.207818 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.207991 1856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:25.208096 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "a91d1f34c6d64ca38294c090bf04f977"
format_stamp: "Formatted at 2026-06-20 14:31:25 on dist-test-slave-tc2g"
server_key: "d7ad0b2ec572d58411a8569bf626c504"
server_key_iv: "dd4b0311b30eff3dc5d7b674b349f6b7"
server_key_version: "encryptionkey@0"
I20260620 14:31:25.208178 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:25.225046 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:25.227329 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:25.227484 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:25.228047 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:25.228104 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.228147 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:25.228190 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:25.235248 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41855
I20260620 14:31:25.235584 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:25.235550 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41855 every 8 connection(s)
I20260620 14:31:25.237818 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.228488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36741 (local address 127.0.98.62:33267)
0620 14:31:25.228533 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.228535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.228731 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:31:25.229261 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.229263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.229275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.229310 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.230036 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.230529 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.231624 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.231881 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.231963 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.234990 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.234992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.234992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.235001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.236486 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.236878 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.236880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.236880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.236939 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.237182 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.237183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.237184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.237290 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.237361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.237633 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.237703 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:31:25.237980 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.228478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.3:36741)
0620 14:31:25.228614 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.228646 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.229153 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.229396 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.229400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.229752 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.229932 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.229937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.230627 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.230628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.231526 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.231528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.231969 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.231971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.232097 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.232593 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.232613 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.234889 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:25.236579 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.236585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.236586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.236791 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.237002 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.237002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.237003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.237120 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.237352 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:25.237357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:25.237504 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.237613 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.237749 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260620 14:31:25.238461 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33267
I20260620 14:31:25.238548 1987 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:25.238750 1987 heartbeater.cc:511] Master 127.0.98.62:33267 requested a full tablet report, sending...
I20260620 14:31:25.239176 1503 ts_manager.cc:194] Registered new tserver with Master: a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855)
I20260620 14:31:25.239653 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36741
I20260620 14:31:25.244088 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.239524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36843 (local address 127.0.98.62:33267)
0620 14:31:25.239561 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.239563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.239621 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:31:25.239685 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.239685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.239695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.239746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.240108 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.240698 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.241336 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.241562 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.241611 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.242360 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.242360 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.242361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.242366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.243360 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.243645 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.243646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.243646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.243686 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.243790 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.243790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.243790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.243861 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.243900 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.243991 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.244038 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:25.244100 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.239464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.98.3:36843)
0620 14:31:25.239566 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.239575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.239631 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.239748 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.239749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.239959 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.240038 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.240039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.240776 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.240777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.241268 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.241269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.241622 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.241622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.241689 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.242027 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.242030 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.242270 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:25.243426 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.243427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.243427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.243575 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.243722 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.243722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.243722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.243759 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.243892 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:25.243892 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:25.243918 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.243976 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.244038 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
I20260620 14:31:25.244433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 1856
I20260620 14:31:25.244517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:25.244668 392 external_mini_cluster.cc:1428] Setting key fd872104ef58ffae3b827cb1dc0cef2e
I20260620 14:31:25.245328 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:25.277711 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.270094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.98.62:33267)
0620 14:31:25.270164 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.270166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.270176 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:25.270209 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.270211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.270223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.270307 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.270923 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.271695 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.272688 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.272894 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.272978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.275323 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.275324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.275324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.275330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.276790 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.277082 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.277083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.277083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.277128 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.277291 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.277291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.277291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.277377 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.277506 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.277589 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.277647 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:31:25.279376 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:25.280999 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:25.287232 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:37298)
0620 14:31:25.283498 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.283509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.283558 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.284003 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.284006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.284008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.284102 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.284106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.285037 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.285038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.286258 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.286268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.286946 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.286947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.287028 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.287041 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.287108 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.287154 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260620 14:31:25.287240 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.0.98.1:37167)
0620 14:31:25.283810 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.283814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.283825 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:25.283864 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.283867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.283883 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.283993 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.284231 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.284944 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.286344 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.286855 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.286945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.287029 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.287122 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.287157 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260620 14:31:25.287369 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.0.1:43750)
0620 14:31:25.283815 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.283824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.283876 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.284059 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.284060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.284062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.284122 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.284124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.285492 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.285493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.286316 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.286318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.287151 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.287153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.287210 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.287214 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.287241 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.287279 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260620 14:31:25.287719 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.98.2:36469)
0620 14:31:25.283792 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.283798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.283865 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:31:25.283940 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.283942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.283958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.284195 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.284259 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.285343 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.286395 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.287053 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.287473 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.287553 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.287577 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.287615 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 14:31:25.287774 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41855 (local address 127.0.0.1:60600)
0620 14:31:25.283442 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.283483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.283550 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.284147 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.284148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.284150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.284186 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.284187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.285442 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.285443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.286451 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.286453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.287184 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.287185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.287230 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.287233 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.287259 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.287708 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260620 14:31:25.287833 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 8587bbf109b74789a450fad20ad1063b (DEFAULT_TABLE table=test-table [id=cb215c1e38994a2e9846ade8929542e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.288195 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587bbf109b74789a450fad20ad1063b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.288293 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.283831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.98.3:41855)
0620 14:31:25.283995 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.283999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.284008 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:25.284042 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.284043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.284057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.284262 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.284339 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.285338 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.286518 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.287110 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.288034 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.288108 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.288131 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.288166 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260620 14:31:25.288357 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 8587bbf109b74789a450fad20ad1063b (DEFAULT_TABLE table=test-table [id=cb215c1e38994a2e9846ade8929542e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.288638 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587bbf109b74789a450fad20ad1063b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.291517 2010 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Bootstrap starting.
I20260620 14:31:25.292179 2010 tablet_bootstrap.cc:654] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.292109 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 8587bbf109b74789a450fad20ad1063b (DEFAULT_TABLE table=test-table [id=cb215c1e38994a2e9846ade8929542e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.292448 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587bbf109b74789a450fad20ad1063b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.292511 2010 log.cc:824] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:25.292830 2011 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Bootstrap starting.
I20260620 14:31:25.293192 2010 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: No bootstrap required, opened a new log
I20260620 14:31:25.293298 2010 ts_tablet_manager.cc:1397] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:25.294003 2011 tablet_bootstrap.cc:654] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.294315 2011 log.cc:824] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:25.295015 2010 raft_consensus.cc:348] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.295106 2011 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: No bootstrap required, opened a new log
I20260620 14:31:25.295161 2010 raft_consensus.cc:374] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.295183 2011 ts_tablet_manager.cc:1397] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:25.295203 2010 raft_consensus.cc:729] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44234692039f4f5b82180b9ba605fd8f, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.295334 2010 consensus_queue.cc:260] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.295599 2010 ts_tablet_manager.cc:1428] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:25.295616 1707 heartbeater.cc:503] Master 127.0.98.62:33267 was elected leader, sending a full tablet report...
I20260620 14:31:25.296216 2015 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Bootstrap starting.
I20260620 14:31:25.296844 2015 tablet_bootstrap.cc:654] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.297168 2015 log.cc:824] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:25.297511 2011 raft_consensus.cc:348] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.297667 2011 raft_consensus.cc:374] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.297717 2011 raft_consensus.cc:729] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f09eb0be24494da2ecb899210867e2, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.297837 2011 consensus_queue.cc:260] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.297945 2015 tablet_bootstrap.cc:492] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: No bootstrap required, opened a new log
I20260620 14:31:25.298041 2015 ts_tablet_manager.cc:1397] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:25.298184 2011 ts_tablet_manager.cc:1428] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:25.298244 1847 heartbeater.cc:503] Master 127.0.98.62:33267 was elected leader, sending a full tablet report...
I20260620 14:31:25.299417 2015 raft_consensus.cc:348] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.299528 2015 raft_consensus.cc:374] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.299549 2015 raft_consensus.cc:729] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91d1f34c6d64ca38294c090bf04f977, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.299672 2015 consensus_queue.cc:260] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.299914 2015 ts_tablet_manager.cc:1428] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:25.299969 1987 heartbeater.cc:503] Master 127.0.98.62:33267 was elected leader, sending a full tablet report...
W20260620 14:31:25.330467 1848 tablet.cc:2369] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:25.383319 2018 raft_consensus.cc:482] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:25.383488 2018 raft_consensus.cc:504] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.383853 2018 leader_election.cc:288] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167), f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469)
I20260620 14:31:25.386631 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.384048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.98.3:34397)
0620 14:31:25.384108 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.384119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.384166 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.384348 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.384349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.384351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.384428 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.384429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.385051 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.385053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.385789 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.385791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.386354 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.386355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.386438 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.386441 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.386513 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.386553 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:31:25.386620 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.384098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34397 (local address 127.0.98.1:37167)
0620 14:31:25.384152 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.384154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.384164 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:25.384234 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.384234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.384246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.384307 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.384512 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.384964 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.385899 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.386269 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.386363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.386373 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.386520 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.386556 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:31:25.386883 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.384115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58615 (local address 127.0.98.2:36469)
0620 14:31:25.384159 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.384161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.384416 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:31:25.384505 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.384505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.384517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.384571 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.384722 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.385161 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.386098 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.386480 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.386571 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.386579 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.386756 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.386812 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:25.386974 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587bbf109b74789a450fad20ad1063b" candidate_uuid: "a91d1f34c6d64ca38294c090bf04f977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44234692039f4f5b82180b9ba605fd8f" is_pre_election: true
I20260620 14:31:25.387096 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.384048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.98.3:58615)
0620 14:31:25.384311 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.384319 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.384416 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.384575 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.384575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.384576 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.384653 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.384654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.385250 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.385251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.386010 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.386011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.386574 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.386575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.386648 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.386651 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.386966 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.387006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":125,"thread_start_us":120,"threads_started":1}
I20260620 14:31:25.387130 1661 raft_consensus.cc:2454] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91d1f34c6d64ca38294c090bf04f977 in term 0.
I20260620 14:31:25.387363 1875 leader_election.cc:302] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, a91d1f34c6d64ca38294c090bf04f977; no voters:
I20260620 14:31:25.387382 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587bbf109b74789a450fad20ad1063b" candidate_uuid: "a91d1f34c6d64ca38294c090bf04f977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f09eb0be24494da2ecb899210867e2" is_pre_election: true
I20260620 14:31:25.387511 2018 raft_consensus.cc:2781] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:25.387529 1801 raft_consensus.cc:2454] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91d1f34c6d64ca38294c090bf04f977 in term 0.
I20260620 14:31:25.387571 2018 raft_consensus.cc:482] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:25.387604 2018 raft_consensus.cc:3037] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.388473 2018 raft_consensus.cc:504] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.388618 2018 leader_election.cc:288] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [CANDIDATE]: Term 1 election: Requested vote from peers 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167), f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469)
I20260620 14:31:25.388761 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587bbf109b74789a450fad20ad1063b" candidate_uuid: "a91d1f34c6d64ca38294c090bf04f977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44234692039f4f5b82180b9ba605fd8f"
I20260620 14:31:25.388846 1661 raft_consensus.cc:3037] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.388875 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587bbf109b74789a450fad20ad1063b" candidate_uuid: "a91d1f34c6d64ca38294c090bf04f977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f09eb0be24494da2ecb899210867e2"
I20260620 14:31:25.388955 1801 raft_consensus.cc:3037] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.389642 1661 raft_consensus.cc:2454] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91d1f34c6d64ca38294c090bf04f977 in term 1.
I20260620 14:31:25.389760 1801 raft_consensus.cc:2454] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91d1f34c6d64ca38294c090bf04f977 in term 1.
I20260620 14:31:25.389856 1875 leader_election.cc:302] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, a91d1f34c6d64ca38294c090bf04f977; no voters:
I20260620 14:31:25.389956 2018 raft_consensus.cc:2781] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:25.390141 2018 raft_consensus.cc:686] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 1 LEADER]: Becoming Leader. State: Replica: a91d1f34c6d64ca38294c090bf04f977, State: Running, Role: LEADER
I20260620 14:31:25.390228 2018 consensus_queue.cc:237] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.391007 1504 catalog_manager.cc:5495] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a91d1f34c6d64ca38294c090bf04f977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:25.404333 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.402100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.0.98.3:41855)
0620 14:31:25.402161 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.402163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.402184 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:31:25.402260 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.402261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.402277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.402328 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:25.402450 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.402899 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.403611 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.403783 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.403830 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.403995 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.404081 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.404189 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.404218 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.404235 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:31:25.406107 1661 raft_consensus.cc:1261] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Refusing update from remote peer a91d1f34c6d64ca38294c090bf04f977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.406141 1801 raft_consensus.cc:1261] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Refusing update from remote peer a91d1f34c6d64ca38294c090bf04f977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.406450 2020 consensus_queue.cc:1035] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.406566 2018 consensus_queue.cc:1035] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.409538 2024 mvcc.cc:205] Tried to move back new op lower bound from 7298932266620899328 to 7298932266559062016. Current Snapshot: MvccSnapshot[applied={T|T < 7298932266620899328 or (T in {7298932266620899328})}]
I20260620 14:31:25.409541 2025 mvcc.cc:205] Tried to move back new op lower bound from 7298932266620899328 to 7298932266559062016. Current Snapshot: MvccSnapshot[applied={T|T < 7298932266620899328}]
I20260620 14:31:25.412281 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:25.412381 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:25.413233 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Sending DeleteTablet for 3 replicas of tablet 8587bbf109b74789a450fad20ad1063b
I20260620 14:31:25.413552 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 8587bbf109b74789a450fad20ad1063b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37298
I20260620 14:31:25.413569 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8587bbf109b74789a450fad20ad1063b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60600
I20260620 14:31:25.413632 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 8587bbf109b74789a450fad20ad1063b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260620 14:31:25.413929 2032 tablet_replica.cc:330] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: stopping tablet replica
I20260620 14:31:25.414029 2032 raft_consensus.cc:2229] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:25.414043 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:31:25.414155 2032 raft_consensus.cc:2258] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:31:25.414182 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:25.414304 2033 tablet_replica.cc:330] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: stopping tablet replica
I20260620 14:31:25.414537 2033 raft_consensus.cc:2229] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:25.414650 2033 raft_consensus.cc:2258] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:25.415148 2033 ts_tablet_manager.cc:1905] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.415585 1921 tablet_service.cc:1467] Processing CreateTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 (DEFAULT_TABLE table=test-table [id=af052f30301c4bb1b1c2b87a31ca3280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.415688 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88ce2492b4d4f169965e2a5fc66a1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.415849 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 (DEFAULT_TABLE table=test-table [id=af052f30301c4bb1b1c2b87a31ca3280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.415964 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88ce2492b4d4f169965e2a5fc66a1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.416786 2034 tablet_replica.cc:330] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: stopping tablet replica
I20260620 14:31:25.416932 2034 raft_consensus.cc:2229] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:25.417019 2033 ts_tablet_manager.cc:1918] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:25.417038 2034 raft_consensus.cc:2258] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:25.417086 2033 log.cc:1196] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/8587bbf109b74789a450fad20ad1063b
I20260620 14:31:25.417094 2015 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Bootstrap starting.
I20260620 14:31:25.417410 2033 ts_tablet_manager.cc:1939] T 8587bbf109b74789a450fad20ad1063b P 44234692039f4f5b82180b9ba605fd8f: Deleting consensus metadata
I20260620 14:31:25.417517 2015 tablet_bootstrap.cc:654] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.417768 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 (DEFAULT_TABLE table=test-table [id=af052f30301c4bb1b1c2b87a31ca3280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:25.417829 2011 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Bootstrap starting.
I20260620 14:31:25.417860 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88ce2492b4d4f169965e2a5fc66a1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.418141 2032 ts_tablet_manager.cc:1905] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.418242 2011 tablet_bootstrap.cc:654] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.418852 2011 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: No bootstrap required, opened a new log
I20260620 14:31:25.418846 2015 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: No bootstrap required, opened a new log
I20260620 14:31:25.418900 2011 ts_tablet_manager.cc:1397] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:25.418927 2015 ts_tablet_manager.cc:1397] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:31:25.418972 2010 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Bootstrap starting.
I20260620 14:31:25.419044 2011 raft_consensus.cc:348] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.419057 2015 raft_consensus.cc:348] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.419123 2011 raft_consensus.cc:374] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.419129 2015 raft_consensus.cc:374] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.419150 2015 raft_consensus.cc:729] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91d1f34c6d64ca38294c090bf04f977, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.419149 2011 raft_consensus.cc:729] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f09eb0be24494da2ecb899210867e2, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.419211 2011 consensus_queue.cc:260] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.419204 2015 consensus_queue.cc:260] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.419291 2011 ts_tablet_manager.cc:1428] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.419312 2015 ts_tablet_manager.cc:1428] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.419596 2010 tablet_bootstrap.cc:654] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.419615 1491 catalog_manager.cc:4841] TS 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167): tablet 8587bbf109b74789a450fad20ad1063b (table test-table [id=cb215c1e38994a2e9846ade8929542e1]) successfully deleted
I20260620 14:31:25.420249 2034 ts_tablet_manager.cc:1905] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.421389 2034 ts_tablet_manager.cc:1918] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:25.421499 2034 log.cc:1196] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/8587bbf109b74789a450fad20ad1063b
I20260620 14:31:25.421625 2010 tablet_bootstrap.cc:492] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: No bootstrap required, opened a new log
I20260620 14:31:25.421660 2010 ts_tablet_manager.cc:1397] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:25.421759 2034 ts_tablet_manager.cc:1939] T 8587bbf109b74789a450fad20ad1063b P f6f09eb0be24494da2ecb899210867e2: Deleting consensus metadata
I20260620 14:31:25.421777 2010 raft_consensus.cc:348] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.421819 2010 raft_consensus.cc:374] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.421831 2010 raft_consensus.cc:729] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44234692039f4f5b82180b9ba605fd8f, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.421890 2010 consensus_queue.cc:260] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.421993 2010 ts_tablet_manager.cc:1428] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.422267 1489 catalog_manager.cc:4841] TS f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469): tablet 8587bbf109b74789a450fad20ad1063b (table test-table [id=cb215c1e38994a2e9846ade8929542e1]) successfully deleted
I20260620 14:31:25.426380 2032 ts_tablet_manager.cc:1918] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:25.426478 2032 log.cc:1196] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/8587bbf109b74789a450fad20ad1063b
I20260620 14:31:25.426780 2032 ts_tablet_manager.cc:1939] T 8587bbf109b74789a450fad20ad1063b P a91d1f34c6d64ca38294c090bf04f977: Deleting consensus metadata
I20260620 14:31:25.427189 1490 catalog_manager.cc:4841] TS a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855): tablet 8587bbf109b74789a450fad20ad1063b (table test-table [id=cb215c1e38994a2e9846ade8929542e1]) successfully deleted
W20260620 14:31:25.486116 1988 tablet.cc:2369] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:25.661912 1708 tablet.cc:2369] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:25.666196 2017 raft_consensus.cc:482] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:25.666281 2017 raft_consensus.cc:504] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.666643 2017 leader_election.cc:288] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167), a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855)
I20260620 14:31:25.670619 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.666971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41625 (local address 127.0.98.3:41855)
0620 14:31:25.667011 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.667013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.667063 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:31:25.667145 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.667146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.667156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.667212 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.667430 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.668160 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.669719 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.670266 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.670346 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.670356 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.670511 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.670550 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:25.670619 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.666920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59189 (local address 127.0.98.1:37167)
0620 14:31:25.666970 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.666972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.667066 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:31:25.667145 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.667146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.667155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.667212 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.667430 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.668160 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.669719 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.670266 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.670345 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.670356 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.670512 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.670549 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:25.670655 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.666846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41855 (local address 127.0.98.2:41625)
0620 14:31:25.666981 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.666994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.667064 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.667236 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.667237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.667240 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.667342 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.667345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.668293 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.668293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.669627 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.669631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.670356 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.670357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.670430 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.670435 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.670493 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.670545 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":68,"thread_start_us":70,"threads_started":1}
I20260620 14:31:25.670655 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.666846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.98.2:59189)
0620 14:31:25.666999 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.667007 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.667064 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.667236 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.667237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.667240 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.667342 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.667346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.668292 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.668292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.669627 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.669631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.670363 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.670364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.670430 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.670434 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.670493 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.670545 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":122,"thread_start_us":40,"threads_started":1}
I20260620 14:31:25.671082 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88ce2492b4d4f169965e2a5fc66a1d6" candidate_uuid: "f6f09eb0be24494da2ecb899210867e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91d1f34c6d64ca38294c090bf04f977" is_pre_election: true
I20260620 14:31:25.671124 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88ce2492b4d4f169965e2a5fc66a1d6" candidate_uuid: "f6f09eb0be24494da2ecb899210867e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44234692039f4f5b82180b9ba605fd8f" is_pre_election: true
I20260620 14:31:25.671203 1661 raft_consensus.cc:2454] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f09eb0be24494da2ecb899210867e2 in term 0.
I20260620 14:31:25.671231 1941 raft_consensus.cc:2454] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f09eb0be24494da2ecb899210867e2 in term 0.
I20260620 14:31:25.671419 1735 leader_election.cc:302] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, f6f09eb0be24494da2ecb899210867e2; no voters:
I20260620 14:31:25.671537 2017 raft_consensus.cc:2781] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:25.671576 2017 raft_consensus.cc:482] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:25.671624 2017 raft_consensus.cc:3037] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.672384 2017 raft_consensus.cc:504] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.672533 2017 leader_election.cc:288] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [CANDIDATE]: Term 1 election: Requested vote from peers 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167), a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855)
I20260620 14:31:25.672691 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88ce2492b4d4f169965e2a5fc66a1d6" candidate_uuid: "f6f09eb0be24494da2ecb899210867e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44234692039f4f5b82180b9ba605fd8f"
I20260620 14:31:25.672720 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88ce2492b4d4f169965e2a5fc66a1d6" candidate_uuid: "f6f09eb0be24494da2ecb899210867e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91d1f34c6d64ca38294c090bf04f977"
I20260620 14:31:25.672785 1661 raft_consensus.cc:3037] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.672796 1941 raft_consensus.cc:3037] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.673561 1661 raft_consensus.cc:2454] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f09eb0be24494da2ecb899210867e2 in term 1.
I20260620 14:31:25.673650 1941 raft_consensus.cc:2454] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f09eb0be24494da2ecb899210867e2 in term 1.
I20260620 14:31:25.673759 1735 leader_election.cc:302] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, f6f09eb0be24494da2ecb899210867e2; no voters:
I20260620 14:31:25.673856 2017 raft_consensus.cc:2781] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:25.674026 2017 raft_consensus.cc:686] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 1 LEADER]: Becoming Leader. State: Replica: f6f09eb0be24494da2ecb899210867e2, State: Running, Role: LEADER
I20260620 14:31:25.674103 2017 consensus_queue.cc:237] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.674713 1503 catalog_manager.cc:5495] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f09eb0be24494da2ecb899210867e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f6f09eb0be24494da2ecb899210867e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:25.728482 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.722108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.98.62:33267)
0620 14:31:25.722280 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.722282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.722291 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:25.722314 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.722315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.722323 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.722382 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.722790 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.723254 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.723886 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.724051 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.724097 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.726140 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.726142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.726142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.726148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.727518 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.727874 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.727875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.727875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.727922 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.728086 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.728087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.728087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.728192 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.728241 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.728357 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.728420 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260620 14:31:25.728616 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.722055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33267 (local address 127.0.0.1:43094)
0620 14:31:25.722214 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.722223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.722272 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.722399 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.722400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.722680 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.722724 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.722726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.723316 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.723316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.723804 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.723805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.724109 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.724110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.724145 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.724598 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.724602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.726062 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:25.727589 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.727594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.727596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.727777 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.727979 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:25.727980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:25.727980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:25.728035 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:25.728245 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:25.728250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:25.728289 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.728341 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.728466 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":46,"thread_start_us":69,"threads_started":1}
I20260620 14:31:25.730611 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:31:25.730826 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:25.732227 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 60bab37b034b413cacaefc3cb1ade85d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec6715c6310d48cebaca0f93c50a8b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:25.732218 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 60bab37b034b413cacaefc3cb1ade85d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec6715c6310d48cebaca0f93c50a8b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:25.732398 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bab37b034b413cacaefc3cb1ade85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.732398 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bab37b034b413cacaefc3cb1ade85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.733996 2010 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: Bootstrap starting.
I20260620 14:31:25.733997 2011 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: Bootstrap starting.
I20260620 14:31:25.734072 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 60bab37b034b413cacaefc3cb1ade85d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec6715c6310d48cebaca0f93c50a8b3a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:25.734211 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bab37b034b413cacaefc3cb1ade85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:25.734467 2010 tablet_bootstrap.cc:654] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.734467 2011 tablet_bootstrap.cc:654] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.735137 2010 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: No bootstrap required, opened a new log
I20260620 14:31:25.735204 2010 ts_tablet_manager.cc:1397] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:25.735359 2010 raft_consensus.cc:348] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.735437 2010 raft_consensus.cc:374] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.735468 2010 raft_consensus.cc:729] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44234692039f4f5b82180b9ba605fd8f, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.735471 2015 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: Bootstrap starting.
I20260620 14:31:25.735517 2010 consensus_queue.cc:260] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.735630 2010 ts_tablet_manager.cc:1428] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:25.735821 2015 tablet_bootstrap.cc:654] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:25.735780 2014 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.735883 2014 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.735934 2011 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: No bootstrap required, opened a new log
I20260620 14:31:25.735975 2011 ts_tablet_manager.cc:1397] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:25.736116 2011 raft_consensus.cc:348] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.736199 2011 raft_consensus.cc:374] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.736231 2011 raft_consensus.cc:729] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f09eb0be24494da2ecb899210867e2, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.736277 2011 consensus_queue.cc:260] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.736392 2011 ts_tablet_manager.cc:1428] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.736438 2015 tablet_bootstrap.cc:492] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: No bootstrap required, opened a new log
I20260620 14:31:25.736456 2017 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.736488 2015 ts_tablet_manager.cc:1397] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:25.736521 2017 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.736590 2015 raft_consensus.cc:348] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.736660 2015 raft_consensus.cc:374] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:25.736686 2015 raft_consensus.cc:729] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91d1f34c6d64ca38294c090bf04f977, State: Initialized, Role: FOLLOWER
I20260620 14:31:25.736734 2015 consensus_queue.cc:260] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.736828 2015 ts_tablet_manager.cc:1428] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:25.736871 2018 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.736956 2018 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.877209 2014 raft_consensus.cc:482] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:25.877312 2014 raft_consensus.cc:504] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.877725 2014 leader_election.cc:288] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469), a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855)
I20260620 14:31:25.881631 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.877984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33115 (local address 127.0.98.2:36469)
0620 14:31:25.878034 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.878036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.878121 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:31:25.878201 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.878203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.878214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.878359 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.878577 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.879328 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.880679 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.881247 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.881323 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.881334 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.881522 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.881558 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:31:25.881666 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.878074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47883 (local address 127.0.98.3:41855)
0620 14:31:25.878133 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.878135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.878144 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:25.878347 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.878348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.878361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.878432 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:25.878634 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.879266 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.880694 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.881295 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.881373 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.881384 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.881560 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.881597 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:31:25.881680 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.877922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.98.1:33115)
0620 14:31:25.878047 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.878058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.878128 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.878381 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.878382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.878384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.878483 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.878485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.879440 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.879442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.880540 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.880542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.881333 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.881334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.881404 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.881407 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.881515 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.881556 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":23,"thread_start_us":62,"threads_started":1}
I20260620 14:31:25.881690 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.877955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41855 (local address 127.0.98.1:47883)
0620 14:31:25.878085 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.878096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.878158 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.878447 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.878447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.878449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:25.878532 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.878533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.879411 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.879412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.880530 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.880532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.881371 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.881372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.881465 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.881468 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.881552 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.881592 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":17,"thread_start_us":78,"threads_started":1}
I20260620 14:31:25.881888 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bab37b034b413cacaefc3cb1ade85d" candidate_uuid: "44234692039f4f5b82180b9ba605fd8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f09eb0be24494da2ecb899210867e2" is_pre_election: true
I20260620 14:31:25.881994 1801 raft_consensus.cc:2454] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44234692039f4f5b82180b9ba605fd8f in term 0.
I20260620 14:31:25.882143 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bab37b034b413cacaefc3cb1ade85d" candidate_uuid: "44234692039f4f5b82180b9ba605fd8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91d1f34c6d64ca38294c090bf04f977" is_pre_election: true
I20260620 14:31:25.882169 1593 leader_election.cc:302] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, f6f09eb0be24494da2ecb899210867e2; no voters:
I20260620 14:31:25.882230 1941 raft_consensus.cc:2454] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44234692039f4f5b82180b9ba605fd8f in term 0.
I20260620 14:31:25.882283 2014 raft_consensus.cc:2781] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:25.882323 2014 raft_consensus.cc:482] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:25.882349 2014 raft_consensus.cc:3037] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.883051 2014 raft_consensus.cc:504] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.883193 2014 leader_election.cc:288] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [CANDIDATE]: Term 1 election: Requested vote from peers f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469), a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855)
I20260620 14:31:25.883373 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bab37b034b413cacaefc3cb1ade85d" candidate_uuid: "44234692039f4f5b82180b9ba605fd8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91d1f34c6d64ca38294c090bf04f977"
I20260620 14:31:25.883448 1941 raft_consensus.cc:3037] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.884055 1941 raft_consensus.cc:2454] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44234692039f4f5b82180b9ba605fd8f in term 1.
I20260620 14:31:25.884215 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bab37b034b413cacaefc3cb1ade85d" candidate_uuid: "44234692039f4f5b82180b9ba605fd8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f09eb0be24494da2ecb899210867e2"
I20260620 14:31:25.884241 1594 leader_election.cc:302] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234692039f4f5b82180b9ba605fd8f, a91d1f34c6d64ca38294c090bf04f977; no voters:
I20260620 14:31:25.884289 1801 raft_consensus.cc:3037] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:25.884369 2014 raft_consensus.cc:2781] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:25.884507 2014 raft_consensus.cc:686] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [term 1 LEADER]: Becoming Leader. State: Replica: 44234692039f4f5b82180b9ba605fd8f, State: Running, Role: LEADER
I20260620 14:31:25.884581 2014 consensus_queue.cc:237] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } }
I20260620 14:31:25.884936 2053 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: TxnStatusTablet state changed. Reason: New leader 44234692039f4f5b82180b9ba605fd8f. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.885084 2053 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:25.885149 1801 raft_consensus.cc:2454] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44234692039f4f5b82180b9ba605fd8f in term 1.
I20260620 14:31:25.885370 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:31:25.885305 1502 catalog_manager.cc:5495] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 44234692039f4f5b82180b9ba605fd8f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:25.885532 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:31:25.922039 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.919294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.0.98.1:37167)
0620 14:31:25.919335 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.919337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.919383 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:31:25.919490 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.919491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.919510 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.919587 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:25.919720 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.920184 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.921260 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.921502 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.921580 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.921750 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.921831 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.921878 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.921954 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.921986 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:25.922094 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.919252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:37314)
0620 14:31:25.919340 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.919351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.919402 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.919604 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.919605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.919606 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:25.919657 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.919660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.920254 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.920254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.921178 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.921180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.921587 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.921588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.921642 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.921658 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.921889 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.921896 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.921951 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.922013 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:31:25.923254 1801 raft_consensus.cc:1261] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Refusing update from remote peer 44234692039f4f5b82180b9ba605fd8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.923398 1941 raft_consensus.cc:1261] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Refusing update from remote peer 44234692039f4f5b82180b9ba605fd8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.923518 2014 consensus_queue.cc:1035] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.923671 2053 consensus_queue.cc:1035] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.924021 2017 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: TxnStatusTablet state changed. Reason: New leader 44234692039f4f5b82180b9ba605fd8f. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.924124 2017 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.924401 2018 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: TxnStatusTablet state changed. Reason: New leader 44234692039f4f5b82180b9ba605fd8f. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.924532 2018 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.925199 2014 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.925231 2053 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.925293 2014 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:25.925300 2053 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P 44234692039f4f5b82180b9ba605fd8f: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:25.925498 2017 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.925578 2017 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P f6f09eb0be24494da2ecb899210867e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.926532 2061 mvcc.cc:205] Tried to move back new op lower bound from 7298932268739559424 to 7298932268583714816. Current Snapshot: MvccSnapshot[applied={T|T < 7298932268739559424}]
I20260620 14:31:25.926617 2018 tablet_replica.cc:439] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44234692039f4f5b82180b9ba605fd8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "f6f09eb0be24494da2ecb899210867e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } } peers { permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 } } }
I20260620 14:31:25.926694 2018 tablet_replica.cc:442] T 60bab37b034b413cacaefc3cb1ade85d P a91d1f34c6d64ca38294c090bf04f977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:25.931926 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.926981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.0.98.2:36469)
0620 14:31:25.927022 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.927024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.929600 (+ 2576us) server_negotiation.cc:407] Connection header received
0620 14:31:25.929681 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.929682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.929700 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.929783 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:25.929892 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.930398 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.931150 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.931355 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.931438 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.931607 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.931710 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.931792 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.931817 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.931837 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:25.936866 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.934343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46299 (local address 127.0.98.1:37167)
0620 14:31:25.934387 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.934389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.934476 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:31:25.934551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.934552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.934559 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.934603 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:25.934716 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.935159 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.936247 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.936416 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.936478 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.936669 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.936691 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.936727 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.936804 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.936818 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:31:25.937094 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.934251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.98.2:46299)
0620 14:31:25.934420 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.934432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.934492 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.934612 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.934613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.934615 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:25.934662 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.934663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.935228 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.935229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.936167 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.936169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.936496 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.936497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.936557 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.936570 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.936748 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.936750 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.936802 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.936843 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260620 14:31:25.939383 1661 raft_consensus.cc:1261] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Refusing update from remote peer f6f09eb0be24494da2ecb899210867e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.939381 1941 raft_consensus.cc:1261] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Refusing update from remote peer f6f09eb0be24494da2ecb899210867e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:25.939617 2017 consensus_queue.cc:1035] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44234692039f4f5b82180b9ba605fd8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.939741 2040 consensus_queue.cc:1035] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91d1f34c6d64ca38294c090bf04f977" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:25.947788 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.945408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.98.1:47509)
0620 14:31:25.945579 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.945588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.945661 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.945777 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.945777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.945779 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:25.945820 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.945822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.946434 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.946435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.947013 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.947016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.947437 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.947437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.947498 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.947510 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.947667 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.947668 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:31:25.947697 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:31:25.947726 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":103,"threads_started":1}
I20260620 14:31:25.947785 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.945457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47509 (local address 127.0.98.2:36469)
0620 14:31:25.945492 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.945494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.945646 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:31:25.945717 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.945718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.945727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.945766 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:25.945865 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.946338 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.947087 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.947352 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.947437 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.947583 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:25.947614 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:25.947667 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.947699 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.947719 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:31:25.957049 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:25.957170 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:25.957746 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14d7e26e4bcd4b18a5b1bd60f9a64526: Sending DeleteTablet for 3 replicas of tablet c88ce2492b4d4f169965e2a5fc66a1d6
I20260620 14:31:25.958009 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260620 14:31:25.958072 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60600
I20260620 14:31:25.958138 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c88ce2492b4d4f169965e2a5fc66a1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37298
I20260620 14:31:25.961561 2080 tablet_replica.cc:330] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: stopping tablet replica
I20260620 14:31:25.961673 2080 raft_consensus.cc:2229] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:25.961755 2080 raft_consensus.cc:2258] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:25.961938 2079 tablet_replica.cc:330] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: stopping tablet replica
I20260620 14:31:25.962042 2079 raft_consensus.cc:2229] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:25.962121 2080 ts_tablet_manager.cc:1905] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.962217 2079 raft_consensus.cc:2258] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:25.962484 2079 ts_tablet_manager.cc:1905] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.964196 2079 ts_tablet_manager.cc:1918] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:25.964326 2079 log.cc:1196] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/c88ce2492b4d4f169965e2a5fc66a1d6
I20260620 14:31:25.964654 2079 ts_tablet_manager.cc:1939] T c88ce2492b4d4f169965e2a5fc66a1d6 P f6f09eb0be24494da2ecb899210867e2: Deleting consensus metadata
I20260620 14:31:25.964818 2080 ts_tablet_manager.cc:1918] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:25.964880 2080 log.cc:1196] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/c88ce2492b4d4f169965e2a5fc66a1d6
I20260620 14:31:25.965101 1489 catalog_manager.cc:4841] TS f6f09eb0be24494da2ecb899210867e2 (127.0.98.2:36469): tablet c88ce2492b4d4f169965e2a5fc66a1d6 (table test-table [id=af052f30301c4bb1b1c2b87a31ca3280]) successfully deleted
I20260620 14:31:25.965123 2081 tablet_replica.cc:330] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: stopping tablet replica
I20260620 14:31:25.965185 2081 raft_consensus.cc:2229] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:25.965236 2081 raft_consensus.cc:2258] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:25.965498 2081 ts_tablet_manager.cc:1905] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:25.967273 2081 ts_tablet_manager.cc:1918] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:25.967403 2081 log.cc:1196] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/c88ce2492b4d4f169965e2a5fc66a1d6
I20260620 14:31:25.967713 2081 ts_tablet_manager.cc:1939] T c88ce2492b4d4f169965e2a5fc66a1d6 P 44234692039f4f5b82180b9ba605fd8f: Deleting consensus metadata
Jun 20 14:31:25 dist-test-slave-tc2g krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:25.967868 2080 ts_tablet_manager.cc:1939] T c88ce2492b4d4f169965e2a5fc66a1d6 P a91d1f34c6d64ca38294c090bf04f977: Deleting consensus metadata
I20260620 14:31:25.968122 1491 catalog_manager.cc:4841] TS 44234692039f4f5b82180b9ba605fd8f (127.0.98.1:37167): tablet c88ce2492b4d4f169965e2a5fc66a1d6 (table test-table [id=af052f30301c4bb1b1c2b87a31ca3280]) successfully deleted
I20260620 14:31:25.968279 1490 catalog_manager.cc:4841] TS a91d1f34c6d64ca38294c090bf04f977 (127.0.98.3:41855): tablet c88ce2492b4d4f169965e2a5fc66a1d6 (table test-table [id=af052f30301c4bb1b1c2b87a31ca3280]) successfully deleted
I20260620 14:31:25.971637 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.958967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.1:37167)
0620 14:31:25.959009 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.959011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.959160 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:31:25.959235 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.959236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.959246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.959296 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.960395 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.960987 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.963995 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.964320 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.964384 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.968719 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.968726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.968727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.968734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.970384 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.970769 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.970772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.970773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.970844 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.971047 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.971048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.971048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.971183 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.971320 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.971395 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.971500 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:31:25.971872 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324
I20260620 14:31:25.978509 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.972741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.98.62:33267)
0620 14:31:25.972796 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.972798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.972828 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:31:25.972896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.972897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.972907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.972962 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.973445 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.973916 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.974574 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.974770 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.974840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.975944 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:25.975945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:25.975946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:25.975953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:25.977343 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.977724 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.977725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.977725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.977780 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:25.977989 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:25.977990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:25.977990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:25.978104 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:25.978221 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:25.978312 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:31:25.978384 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 14:31:25.978703 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43110
I20260620 14:31:25.979475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1576
I20260620 14:31:25.986892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1716
I20260620 14:31:25.991493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1856
I20260620 14:31:25.996726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 1472
2026-06-20T14:31:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2537 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:26 dist-test-slave-tc2g krb5kdc[2088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:26 dist-test-slave-tc2g krb5kdc[2088](info): set up 2 sockets
Jun 20 14:31:26 dist-test-slave-tc2g krb5kdc[2088](info): commencing operation
krb5kdc: starting...
W20260620 14:31:27.029130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:27.139926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:27.219026 2101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.219234 2101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.219256 2101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.220335 2101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:27.220368 2101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:31:27.220381 2101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.220394 2101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.220407 2101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:27.220418 2101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:27.221810 2101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46445
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:27.222007 2101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.222175 2101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.226859 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.226859 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.227095 2101 server_base.cc:1034] running on GCE node
W20260620 14:31:27.226884 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.227406 2101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.227681 2101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.228837 2101 hybrid_clock.cc:648] HybridClock initialized: now 1781965887228813 us; error 41 us; skew 500 ppm
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:27.231231 2101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:27.231699 2101 webserver.cc:460] Webserver started at http://127.0.98.62:46017/ using document root <none> and password file <none>
I20260620 14:31:27.231925 2101 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.231969 2101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.232123 2101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.233170 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "ea3e765455ba484ead86f6bb1c0cfb0e"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "df3ac6fba7eafb6adaac11b7dd535f2f"
server_key_iv: "52e5a7fec49d1a5921c61389b88d41fa"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.233554 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea3e765455ba484ead86f6bb1c0cfb0e"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "df3ac6fba7eafb6adaac11b7dd535f2f"
server_key_iv: "52e5a7fec49d1a5921c61389b88d41fa"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.234850 2101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:27.235553 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.235743 2101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:27.235834 2101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "ea3e765455ba484ead86f6bb1c0cfb0e"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "df3ac6fba7eafb6adaac11b7dd535f2f"
server_key_iv: "52e5a7fec49d1a5921c61389b88d41fa"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.235914 2101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.261965 2101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.263778 2101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.263940 2101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.268764 2101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46445
I20260620 14:31:27.268810 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46445 every 8 connection(s)
I20260620 14:31:27.269137 2101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:27.269807 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.272058 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Bootstrap starting.
I20260620 14:31:27.272692 2179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.272989 2179 log.cc:824] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:27.273684 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: No bootstrap required, opened a new log
I20260620 14:31:27.275094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2101
I20260620 14:31:27.275213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:27.275394 392 external_mini_cluster.cc:1428] Setting key f510ecd18dc0d140f0863b9df7797505
I20260620 14:31:27.276286 2179 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 14:31:27.276451 2179 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.276515 2179 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3e765455ba484ead86f6bb1c0cfb0e, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.276654 2179 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 14:31:27.276750 2179 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:27.276793 2179 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:27.276845 2179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:27.277369 2179 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 14:31:27.277506 2179 leader_election.cc:302] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3e765455ba484ead86f6bb1c0cfb0e; no voters:
I20260620 14:31:27.277705 2179 leader_election.cc:288] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:27.277735 2184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:27.277841 2184 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [term 1 LEADER]: Becoming Leader. State: Replica: ea3e765455ba484ead86f6bb1c0cfb0e, State: Running, Role: LEADER
I20260620 14:31:27.277961 2179 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:27.277993 2184 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } }
I20260620 14:31:27.278389 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } } }
I20260620 14:31:27.278466 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:27.278708 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3e765455ba484ead86f6bb1c0cfb0e. Latest consensus state: current_term: 1 leader_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e765455ba484ead86f6bb1c0cfb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46445 } } }
I20260620 14:31:27.279038 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e [sys.catalog]: This master's current role is: LEADER
W20260620 14:31:27.279917 2200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:31:27.279973 2200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:31:27.280018 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:27.280157 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:27.281700 2189 catalog_manager.cc:1347] Generated new cluster ID: afc1639a66e546bcabfd2d0eeda38401
I20260620 14:31:27.281754 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.286006 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.275959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.0.98.62:46445)
0620 14:31:27.276152 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.276159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.276183 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:31:27.276279 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.276284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.276325 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.276461 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.277195 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.278437 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.279504 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.279965 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.280412 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.282769 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.282773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.282774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.282785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.284413 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.284770 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.284773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.284773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.284876 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.285096 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.285097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.285097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.285326 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.285536 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.285602 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.285858 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260620 14:31:27.305030 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:27.305733 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:27.314626 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Generated new TSK 0
I20260620 14:31:27.314829 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:27.348521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:27.428488 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.428707 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.428732 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.429971 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.430013 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.430035 2205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:31:27.430087 2205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:27.430119 2205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:27.431594 2205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:27.431833 2205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.432075 2205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.436935 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.437160 2205 server_base.cc:1034] running on GCE node
W20260620 14:31:27.436909 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.436916 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.437497 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.437760 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.438915 2205 hybrid_clock.cc:648] HybridClock initialized: now 1781965887438904 us; error 39 us; skew 500 ppm
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:27.441366 2205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:27.441812 2205 webserver.cc:460] Webserver started at http://127.0.98.1:34785/ using document root <none> and password file <none>
I20260620 14:31:27.442039 2205 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.442107 2205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.442255 2205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.443336 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a7f3fd6a6634fc0b5760701b65c6276"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "4c6d18333bc0e6cbb82b6135bb9539f8"
server_key_iv: "5bd16c4572bfca63221ca6eb508fc427"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.443689 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a7f3fd6a6634fc0b5760701b65c6276"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "4c6d18333bc0e6cbb82b6135bb9539f8"
server_key_iv: "5bd16c4572bfca63221ca6eb508fc427"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.445048 2205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:31:27.445819 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.445986 2205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.446084 2205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "3a7f3fd6a6634fc0b5760701b65c6276"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "4c6d18333bc0e6cbb82b6135bb9539f8"
server_key_iv: "5bd16c4572bfca63221ca6eb508fc427"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.446166 2205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.462966 2205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.464833 2205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.465013 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.465576 2205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.465641 2205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.465680 2205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.465709 2205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.472919 2205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33819
I20260620 14:31:27.473101 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33819 every 8 connection(s)
I20260620 14:31:27.473399 2205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:27.474854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2205
I20260620 14:31:27.474958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:27.475162 392 external_mini_cluster.cc:1428] Setting key 6647321911eacce192014b1f91bf13d2
I20260620 14:31:27.478389 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.466057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47593 (local address 127.0.98.62:46445)
0620 14:31:27.466100 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.466102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.466352 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:31:27.466882 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.466884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.466896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.466944 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.467739 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.468586 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.470250 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.470597 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.470673 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.473506 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.473508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.473508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.473544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.475818 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.476536 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.476538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.476538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.476598 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.476854 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.476855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.476856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.476988 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.477046 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.478188 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.478281 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:27.478086 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.466045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.1:47593)
0620 14:31:27.466237 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.466271 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.466780 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.467012 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.467016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.467348 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.467655 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.467661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.468689 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.468691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.470155 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.470158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.470694 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.470695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.470859 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.471337 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.471349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.473319 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.475904 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.476114 (+ 210us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.476116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.476444 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.476663 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.476663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.476664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.476790 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.477070 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.477075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.477241 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.477609 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.477802 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":114,"threads_started":1}
I20260620 14:31:27.482506 2336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 14:31:27.482607 2336 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.482772 2336 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 14:31:27.483459 2132 ts_manager.cc:194] Registered new tserver with Master: 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819)
I20260620 14:31:27.484431 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47593
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:31:27.491628 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.482278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.1:46337)
0620 14:31:27.483256 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.483267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.483325 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.483450 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.483451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.483633 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.483700 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.483701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.484841 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.484843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.486075 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.486078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.486651 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.486652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.486721 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.487674 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.487680 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.488120 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.490295 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.490296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.490297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.490517 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.490794 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.490795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.490795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.490875 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.491177 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.491178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.491260 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.491379 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.491493 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":77,"threads_started":1}
I20260620 14:31:27.491633 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.482398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46337 (local address 127.0.98.62:46445)
0620 14:31:27.482442 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.482444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.483313 (+ 869us) server_negotiation.cc:407] Connection header received
0620 14:31:27.483382 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.483383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.483394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.483446 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.483779 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.484729 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.486185 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.486524 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.486572 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.488243 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.488245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.488246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.488253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.490199 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.490617 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.490618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.490619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.490717 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.490939 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.490940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.490940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.491093 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.491178 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.491424 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.491526 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:27.508196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:27.588302 2345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.588493 2345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.588516 2345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.589664 2345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.589704 2345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.589725 2345 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:31:27.589773 2345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:27.589800 2345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:27.591241 2345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:27.591470 2345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.591683 2345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.596776 2353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.596942 2345 server_base.cc:1034] running on GCE node
W20260620 14:31:27.596891 2351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.596786 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.597255 2345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.597513 2345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.598657 2345 hybrid_clock.cc:648] HybridClock initialized: now 1781965887598640 us; error 30 us; skew 500 ppm
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:27.601119 2345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:27.601598 2345 webserver.cc:460] Webserver started at http://127.0.98.2:45365/ using document root <none> and password file <none>
I20260620 14:31:27.601828 2345 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.601902 2345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.602036 2345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.603076 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "9851a3bee3bc428486822f321d585606"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "5edcc4ee8ffba9a86cc0f9a9c99dda10"
server_key_iv: "6377a9abed43faf6ce89786e3ba75fae"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.603442 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9851a3bee3bc428486822f321d585606"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "5edcc4ee8ffba9a86cc0f9a9c99dda10"
server_key_iv: "6377a9abed43faf6ce89786e3ba75fae"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.604733 2345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:27.605470 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.605657 2345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.605762 2345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "9851a3bee3bc428486822f321d585606"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "5edcc4ee8ffba9a86cc0f9a9c99dda10"
server_key_iv: "6377a9abed43faf6ce89786e3ba75fae"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.605847 2345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.617784 2345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.620055 2345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.620213 2345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.620690 2345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.620731 2345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.620755 2345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.620802 2345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.627492 2345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33285
I20260620 14:31:27.627562 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33285 every 8 connection(s)
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.627923 2345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:27.630589 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.621197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57761 (local address 127.0.98.62:46445)
0620 14:31:27.621265 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.621268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.621480 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:31:27.621983 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.621984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.621996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.622060 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.622895 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.623707 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.625266 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.625635 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.625708 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.628348 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.628350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.628350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.628358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.629570 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.629925 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.629926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.629926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.629973 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.630142 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.630142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.630143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.630239 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.630285 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.630473 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.630528 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:31:27.630772 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.621176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.2:57761)
0620 14:31:27.621339 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.621376 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.621878 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.622137 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.622141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.622514 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.622784 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.622789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.623806 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.623807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.625163 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.625166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.625735 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.625736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.625834 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.626449 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.626459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.628240 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.629635 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.629640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.629642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.629851 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.630013 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.630014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.630014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.630094 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.630276 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.630278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.630385 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.630460 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.630571 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":107,"threads_started":1}
I20260620 14:31:27.631211 2476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 14:31:27.631289 2476 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.631472 2476 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 14:31:27.631820 2133 ts_manager.cc:194] Registered new tserver with Master: 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285)
I20260620 14:31:27.632275 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57761
I20260620 14:31:27.632982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2345
I20260620 14:31:27.633080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:27.633246 392 external_mini_cluster.cc:1428] Setting key 74f6eec4a5d1838246ead383e3b7f03a
I20260620 14:31:27.638486 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.632180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.2:41263)
0620 14:31:27.632299 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.632311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.632388 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.632516 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.632517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.632725 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.632813 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.632815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.633699 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.633700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.634512 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.634514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.634817 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.634818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.634895 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.635290 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.635296 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.635623 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.637363 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.637364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.637365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.637601 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.637853 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.637854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.637854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.637921 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.638241 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.638242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.638275 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.638329 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.638415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260620 14:31:27.638602 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.632231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41263 (local address 127.0.98.62:46445)
0620 14:31:27.632397 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.632410 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.632429 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:31:27.632447 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.632447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.632455 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.632513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.632911 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.633583 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.634587 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.634751 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.634802 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.635707 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.635708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.635709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.635717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.637270 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.637685 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.637686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.637687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.637749 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.637986 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.637987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.637987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.638172 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.638248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.638356 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.638436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:27.658216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46445
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:27.737097 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.737323 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.737365 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.738596 2485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.738656 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.738692 2485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:31:27.738739 2485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:27.738770 2485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:27.740248 2485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:27.740511 2485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.740733 2485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.745724 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.745718 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.745886 2485 server_base.cc:1034] running on GCE node
W20260620 14:31:27.745718 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.746155 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.746407 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.747545 2485 hybrid_clock.cc:648] HybridClock initialized: now 1781965887747533 us; error 26 us; skew 500 ppm
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:27.750042 2485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:27.750463 2485 webserver.cc:460] Webserver started at http://127.0.98.3:39647/ using document root <none> and password file <none>
I20260620 14:31:27.750705 2485 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.750748 2485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.750909 2485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.751874 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "909bd0ae228244319ac0958eae66620f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "323ba22f992232d70f891f08e59c49c4"
server_key_iv: "c24f6188967d72d3a753255dea3f0135"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.752199 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "909bd0ae228244319ac0958eae66620f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "323ba22f992232d70f891f08e59c49c4"
server_key_iv: "c24f6188967d72d3a753255dea3f0135"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.753515 2485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:27.754207 2500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.754391 2485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:27.754459 2485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "909bd0ae228244319ac0958eae66620f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-tc2g"
server_key: "323ba22f992232d70f891f08e59c49c4"
server_key_iv: "c24f6188967d72d3a753255dea3f0135"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.754545 2485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.770184 2485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.771957 2485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.772127 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.772715 2485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.772776 2485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.772818 2485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.772842 2485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.780069 2485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42805
I20260620 14:31:27.780187 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42805 every 8 connection(s)
I20260620 14:31:27.780520 2485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:27.782914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2485
I20260620 14:31:27.782933 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.773137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45513 (local address 127.0.98.62:46445)
0620 14:31:27.773211 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.773213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.773436 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:31:27.773971 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.773972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.773986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.774072 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.774667 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.775536 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.777132 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.777496 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.777583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.780194 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.780196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.780196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.780205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.781804 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.782157 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.782158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.782158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.782207 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.782406 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.782406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.782406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.782499 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.782544 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.782752 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.782833 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:31:27.783005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:27.783010 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.773118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.3:45513)
0620 14:31:27.773286 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.773326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.773869 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.774122 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.774125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.774407 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.774592 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.774597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.775646 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.775647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.777020 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.777023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.777599 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.777600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.777747 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.778295 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.778302 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.780041 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.781882 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.781887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.781889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.782095 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.782246 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.782247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.782247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.782343 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.782535 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.782538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.782640 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.782713 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.782824 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":109,"threads_started":1}
I20260620 14:31:27.783180 392 external_mini_cluster.cc:1428] Setting key 18118805b30818fd25a33522cfb663ee
I20260620 14:31:27.783524 2616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46445
I20260620 14:31:27.783627 2616 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.783793 2616 heartbeater.cc:511] Master 127.0.98.62:46445 requested a full tablet report, sending...
I20260620 14:31:27.784183 2132 ts_manager.cc:194] Registered new tserver with Master: 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:27.784595 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45513
I20260620 14:31:27.784914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:27.790810 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.784636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53105 (local address 127.0.98.62:46445)
0620 14:31:27.784672 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.784674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.784886 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:31:27.784967 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.784968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.784979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.785006 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.785379 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.785950 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.786997 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.787188 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.787272 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.788195 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.788196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.788197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.788204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.789773 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.790087 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.790089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.790089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.790150 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.790317 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.790318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.790318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.790450 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.790516 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.790683 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.790751 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:27.790933 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.784579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.98.3:53105)
0620 14:31:27.784806 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.784818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.784888 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.785061 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.785062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.785248 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.785318 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.785319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.786051 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.786052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.786923 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.786926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.787274 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.787275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.787348 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.787777 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.787781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.788088 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.789848 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.789849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.789849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.790012 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.790216 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.790217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.790217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.790264 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.790518 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.790518 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.790571 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.790665 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.790782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.808627 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.801879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.0.98.62:46445)
0620 14:31:27.801924 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.801926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.801936 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:27.802022 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.802023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.802040 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.802103 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.802664 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.803145 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.803901 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.804084 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.804170 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.806322 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.806323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.806323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.806332 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.807511 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.807840 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.807841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.807841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.807920 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.808102 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.808103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.808104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.808214 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.808332 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.808487 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.808563 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:31:27.810184 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:27.811676 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:27.817925 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.813758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.0.1:39682)
0620 14:31:27.814112 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.814121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.814157 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.814321 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.814322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.814325 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:27.814381 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.814383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815854 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.815855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.816803 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.816806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.817605 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.817606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.817682 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.817694 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.817793 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.817843 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260620 14:31:27.817940 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.814002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.0.98.1:33819)
0620 14:31:27.814188 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.814193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.814202 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:27.814235 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.814237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.814249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.814324 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:27.814466 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815564 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.816886 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.817515 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.817604 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.817667 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.817809 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.817849 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260620 14:31:27.818326 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.813757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33285 (local address 127.0.0.1:58310)
0620 14:31:27.813916 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.813961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.814024 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.814287 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.814290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.814292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:27.814356 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.814359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815655 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.815656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.817039 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.817042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.818077 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.818079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.818158 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.818162 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.818224 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.818253 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":57,"thread_start_us":61,"threads_started":1}
I20260620 14:31:27.818392 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.814002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.98.2:33285)
0620 14:31:27.814133 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.814138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.814149 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:27.814190 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.814192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.814209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.814285 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:27.814434 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815557 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.817142 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.817904 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.817974 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.818035 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.818255 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.818296 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260620 14:31:27.818509 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.813757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42805 (local address 127.0.0.1:55248)
0620 14:31:27.814149 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.814160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.814211 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.814696 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.814697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.814698 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:27.814738 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.814739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815718 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.815719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.817040 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.817042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.818119 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.818120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.818169 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.818172 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.818212 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.818251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":68,"spinlock_wait_cycles":768,"thread_start_us":75,"threads_started":1}
I20260620 14:31:27.818951 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.813913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.0.98.3:42805)
0620 14:31:27.814502 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.814505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.814515 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:27.814550 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.814552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.814566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.814633 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:27.814816 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.815577 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.817142 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.817904 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.818273 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.818338 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.818366 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.818850 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":89,"threads_started":1}
I20260620 14:31:27.818907 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 (DEFAULT_TABLE table=test-table [id=c510ba4308cc416db359baaa64c0c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:27.819051 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 (DEFAULT_TABLE table=test-table [id=c510ba4308cc416db359baaa64c0c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:27.819267 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8589dc2bd5497a856512ab6699d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.819322 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8589dc2bd5497a856512ab6699d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.822105 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 (DEFAULT_TABLE table=test-table [id=c510ba4308cc416db359baaa64c0c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:27.822472 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8589dc2bd5497a856512ab6699d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.822932 2636 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Bootstrap starting.
I20260620 14:31:27.823450 2636 tablet_bootstrap.cc:654] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.823741 2636 log.cc:824] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:27.824502 2636 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: No bootstrap required, opened a new log
I20260620 14:31:27.824572 2636 ts_tablet_manager.cc:1397] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:27.825233 2638 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Bootstrap starting.
I20260620 14:31:27.826022 2636 raft_consensus.cc:348] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.826193 2636 raft_consensus.cc:374] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.826234 2638 tablet_bootstrap.cc:654] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.826236 2636 raft_consensus.cc:729] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851a3bee3bc428486822f321d585606, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.826371 2636 consensus_queue.cc:260] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.826546 2638 log.cc:824] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:27.826643 2636 ts_tablet_manager.cc:1428] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:27.826694 2476 heartbeater.cc:503] Master 127.0.98.62:46445 was elected leader, sending a full tablet report...
I20260620 14:31:27.826829 2639 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Bootstrap starting.
I20260620 14:31:27.827278 2638 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: No bootstrap required, opened a new log
I20260620 14:31:27.827347 2638 ts_tablet_manager.cc:1397] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:27.827384 2639 tablet_bootstrap.cc:654] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.827700 2639 log.cc:824] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:27.828537 2639 tablet_bootstrap.cc:492] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: No bootstrap required, opened a new log
I20260620 14:31:27.828617 2639 ts_tablet_manager.cc:1397] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:27.829119 2638 raft_consensus.cc:348] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.829239 2638 raft_consensus.cc:374] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.829260 2638 raft_consensus.cc:729] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.829384 2638 consensus_queue.cc:260] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.829658 2638 ts_tablet_manager.cc:1428] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:27.829692 2616 heartbeater.cc:503] Master 127.0.98.62:46445 was elected leader, sending a full tablet report...
I20260620 14:31:27.830404 2639 raft_consensus.cc:348] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.830603 2639 raft_consensus.cc:374] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.830653 2639 raft_consensus.cc:729] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.830768 2639 consensus_queue.cc:260] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.831038 2639 ts_tablet_manager.cc:1428] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:27.831053 2336 heartbeater.cc:503] Master 127.0.98.62:46445 was elected leader, sending a full tablet report...
W20260620 14:31:27.878575 2477 tablet.cc:2369] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:27.914934 2643 raft_consensus.cc:482] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:27.915053 2643 raft_consensus.cc:504] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.915465 2643 leader_election.cc:288] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819)
I20260620 14:31:27.918346 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.915735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58181 (local address 127.0.98.1:33819)
0620 14:31:27.915785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.915787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.915938 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:31:27.916007 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.916008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.916018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.916080 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:27.916223 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.916696 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.917623 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.917985 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.918065 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.918076 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.918239 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.918274 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:27.918361 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.915777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50141 (local address 127.0.98.2:33285)
0620 14:31:27.915818 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.915820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.915827 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:27.915858 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.915858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.915867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.915943 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:27.916083 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.916591 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.917622 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.918000 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.918092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.918101 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.918262 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.918296 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:27.918370 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.915653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33285 (local address 127.0.98.3:50141)
0620 14:31:27.915711 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.915724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.915772 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.915950 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.915951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.915952 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:27.916019 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.916020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.916679 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.916680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.917540 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.917542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.918093 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.918094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.918181 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.918184 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.918254 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.918294 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":31}
I20260620 14:31:27.918372 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.915673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.98.3:58181)
0620 14:31:27.915859 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.915872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.915946 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.916083 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.916083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.916085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:27.916153 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.916155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.916778 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.916779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.917528 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.917530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.918076 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.918077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.918161 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.918165 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.918222 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.918259 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260620 14:31:27.918807 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8589dc2bd5497a856512ab6699d0d7" candidate_uuid: "909bd0ae228244319ac0958eae66620f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" is_pre_election: true
I20260620 14:31:27.918804 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8589dc2bd5497a856512ab6699d0d7" candidate_uuid: "909bd0ae228244319ac0958eae66620f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606" is_pre_election: true
I20260620 14:31:27.918958 2430 raft_consensus.cc:2454] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bd0ae228244319ac0958eae66620f in term 0.
I20260620 14:31:27.918958 2290 raft_consensus.cc:2454] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909bd0ae228244319ac0958eae66620f in term 0.
I20260620 14:31:27.919186 2503 leader_election.cc:302] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 909bd0ae228244319ac0958eae66620f; no voters:
I20260620 14:31:27.919315 2643 raft_consensus.cc:2781] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:27.919369 2643 raft_consensus.cc:482] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:27.919402 2643 raft_consensus.cc:3037] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:27.920092 2643 raft_consensus.cc:504] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.920233 2643 leader_election.cc:288] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [CANDIDATE]: Term 1 election: Requested vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819)
I20260620 14:31:27.920384 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8589dc2bd5497a856512ab6699d0d7" candidate_uuid: "909bd0ae228244319ac0958eae66620f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606"
I20260620 14:31:27.920392 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b8589dc2bd5497a856512ab6699d0d7" candidate_uuid: "909bd0ae228244319ac0958eae66620f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7f3fd6a6634fc0b5760701b65c6276"
I20260620 14:31:27.920464 2430 raft_consensus.cc:3037] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:27.920464 2290 raft_consensus.cc:3037] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:27.921307 2430 raft_consensus.cc:2454] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bd0ae228244319ac0958eae66620f in term 1.
I20260620 14:31:27.921310 2290 raft_consensus.cc:2454] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909bd0ae228244319ac0958eae66620f in term 1.
I20260620 14:31:27.921490 2501 leader_election.cc:302] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909bd0ae228244319ac0958eae66620f, 9851a3bee3bc428486822f321d585606; no voters:
I20260620 14:31:27.921622 2643 raft_consensus.cc:2781] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:27.921756 2643 raft_consensus.cc:686] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 1 LEADER]: Becoming Leader. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Running, Role: LEADER
I20260620 14:31:27.921871 2643 consensus_queue.cc:237] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } }
I20260620 14:31:27.922734 2133 catalog_manager.cc:5495] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f reported cstate change: term changed from 0 to 1, leader changed from <none> to 909bd0ae228244319ac0958eae66620f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "909bd0ae228244319ac0958eae66620f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:31:27 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:27.952534 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.946140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46445 (local address 127.0.0.1:57312)
0620 14:31:27.946285 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.946295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.946359 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.946489 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.946490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.946781 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.946826 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.946828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.947466 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.947466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.947978 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:27.947979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.948329 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.948329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:27.948370 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.948893 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:27.948898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.950283 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.951644 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.951650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.951651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.951828 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.951990 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:27.951990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:27.951990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:27.952042 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:27.952274 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.952279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.952316 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.952364 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.952445 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260620 14:31:27.952620 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.946213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.0.98.62:46445)
0620 14:31:27.946293 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.946295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.946366 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:31:27.946396 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.946396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.946404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.946473 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.946914 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.947392 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.948062 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:27.948250 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:27.948310 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:27.950356 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.950358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.950358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:27.950364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.951575 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.951889 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.951890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.951890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.951936 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:27.952110 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:27.952110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:27.952111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:27.952217 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.952273 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.952398 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.952491 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":44}
I20260620 14:31:27.954583 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57312:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:31:27.954782 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:27.956261 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 94612692d35b4c62b41572e484863929 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a162accfeeaf4c9686100e7a44d3fbc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:27.956290 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 94612692d35b4c62b41572e484863929 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a162accfeeaf4c9686100e7a44d3fbc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:27.956447 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94612692d35b4c62b41572e484863929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.956460 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94612692d35b4c62b41572e484863929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.957568 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 94612692d35b4c62b41572e484863929 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a162accfeeaf4c9686100e7a44d3fbc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:27.957724 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94612692d35b4c62b41572e484863929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.957870 2636 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: Bootstrap starting.
I20260620 14:31:27.958477 2636 tablet_bootstrap.cc:654] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.959044 2638 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: Bootstrap starting.
I20260620 14:31:27.959059 2639 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: Bootstrap starting.
I20260620 14:31:27.959136 2636 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: No bootstrap required, opened a new log
I20260620 14:31:27.959178 2636 ts_tablet_manager.cc:1397] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:27.959355 2636 raft_consensus.cc:348] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.959443 2639 tablet_bootstrap.cc:654] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.959443 2636 raft_consensus.cc:374] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.959472 2636 raft_consensus.cc:729] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851a3bee3bc428486822f321d585606, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.959481 2638 tablet_bootstrap.cc:654] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.959532 2636 consensus_queue.cc:260] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.959645 2636 ts_tablet_manager.cc:1428] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.959745 2640 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:27.959846 2640 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:27.960162 2638 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: No bootstrap required, opened a new log
I20260620 14:31:27.960219 2638 ts_tablet_manager.cc:1397] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:27.960371 2639 tablet_bootstrap.cc:492] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: No bootstrap required, opened a new log
I20260620 14:31:27.960420 2639 ts_tablet_manager.cc:1397] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:27.960376 2638 raft_consensus.cc:348] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.960456 2638 raft_consensus.cc:374] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.960477 2638 raft_consensus.cc:729] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.960522 2638 consensus_queue.cc:260] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.960573 2639 raft_consensus.cc:348] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.960613 2638 ts_tablet_manager.cc:1428] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.960642 2639 raft_consensus.cc:374] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.960664 2639 raft_consensus.cc:729] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.960652 2643 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:27.960716 2643 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:27.960711 2639 consensus_queue.cc:260] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:27.960815 2639 ts_tablet_manager.cc:1428] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.960866 2644 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:27.961016 2644 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:31:27.978044 2337 tablet.cc:2369] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:28.031404 2617 tablet.cc:2369] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:28.039021 2644 raft_consensus.cc:482] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:28.039130 2644 raft_consensus.cc:504] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.039570 2644 leader_election.cc:288] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:28.042462 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.039797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58667 (local address 127.0.98.2:33285)
0620 14:31:28.039846 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.039849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.039899 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:31:28.039985 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.039986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.039996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.040047 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:28.040206 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.040681 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.041744 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.042121 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.042200 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.042209 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.042353 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.042389 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:31:28.042503 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.039722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33285 (local address 127.0.98.1:58667)
0620 14:31:28.039843 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.039854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.039918 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.040049 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.040050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.040052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:28.040126 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.040128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.040768 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.040769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.041657 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.041659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.042210 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.042211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.042279 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.042283 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.042346 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.042392 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
I20260620 14:31:28.042503 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.039807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51403 (local address 127.0.98.3:42805)
0620 14:31:28.039881 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.039883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.039957 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:31:28.040040 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.040041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.040054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.040122 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:28.040286 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.040744 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.041744 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.042146 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.042232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.042242 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.042395 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.042437 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:31:28.042559 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.039761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42805 (local address 127.0.98.1:51403)
0620 14:31:28.039862 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.039874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.039975 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.040137 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.040137 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.040140 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:28.040213 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.040214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.040830 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.040831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.041659 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.041661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.042233 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.042234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.042305 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.042308 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.042387 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.042427 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260620 14:31:28.042912 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94612692d35b4c62b41572e484863929" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f" is_pre_election: true
I20260620 14:31:28.043045 2570 raft_consensus.cc:2454] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:28.043155 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94612692d35b4c62b41572e484863929" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606" is_pre_election: true
I20260620 14:31:28.043246 2430 raft_consensus.cc:2454] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:28.043432 2222 leader_election.cc:302] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 909bd0ae228244319ac0958eae66620f; no voters:
I20260620 14:31:28.043591 2644 raft_consensus.cc:2781] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:28.043635 2644 raft_consensus.cc:482] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:28.043668 2644 raft_consensus.cc:3037] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.044281 2644 raft_consensus.cc:504] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.044406 2644 leader_election.cc:288] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Requested vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:28.044514 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94612692d35b4c62b41572e484863929" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606"
I20260620 14:31:28.044544 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94612692d35b4c62b41572e484863929" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f"
I20260620 14:31:28.044608 2570 raft_consensus.cc:3037] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.044608 2430 raft_consensus.cc:3037] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.045158 2570 raft_consensus.cc:2454] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:28.045306 2430 raft_consensus.cc:2454] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:28.045312 2222 leader_election.cc:302] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 909bd0ae228244319ac0958eae66620f; no voters:
I20260620 14:31:28.045408 2644 raft_consensus.cc:2781] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:28.045538 2644 raft_consensus.cc:686] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Running, Role: LEADER
I20260620 14:31:28.045621 2644 consensus_queue.cc:237] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.046113 2659 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: TxnStatusTablet state changed. Reason: New leader 3a7f3fd6a6634fc0b5760701b65c6276. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.046195 2659 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:28.046427 2133 catalog_manager.cc:5495] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:28.046732 2661 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:31:28.046819 2661 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:31:28.055080 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.052709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.0.98.1:33819)
0620 14:31:28.052756 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.052758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.052766 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:28.052876 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.052877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.052894 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.052992 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.053115 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.053637 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.054350 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.054539 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.054608 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.054775 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.054868 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.054967 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.055001 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.055020 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:28.055114 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.052651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.0.1:39690)
0620 14:31:28.052722 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.052735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.052784 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.052996 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.052997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.052998 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:28.053042 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.053045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.053714 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.053715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.054257 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.054258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.054608 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.054608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.054676 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.054688 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.054945 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.054954 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.054996 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.055042 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:31:28.056728 2570 raft_consensus.cc:1261] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.056749 2430 raft_consensus.cc:1261] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.057034 2659 consensus_queue.cc:1035] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:28.057184 2644 consensus_queue.cc:1035] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:28.058437 2643 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: TxnStatusTablet state changed. Reason: New leader 3a7f3fd6a6634fc0b5760701b65c6276. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.058528 2643 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:28.058799 2640 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: TxnStatusTablet state changed. Reason: New leader 3a7f3fd6a6634fc0b5760701b65c6276. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.058897 2640 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:28.059255 2659 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.059327 2659 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:28.059764 2640 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.059849 2640 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 9851a3bee3bc428486822f321d585606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:28.059942 2666 mvcc.cc:205] Tried to move back new op lower bound from 7298932277477806080 to 7298932277435543552. Current Snapshot: MvccSnapshot[applied={T|T < 7298932277477806080}]
I20260620 14:31:28.060606 2659 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.060691 2659 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 3a7f3fd6a6634fc0b5760701b65c6276: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:28.060936 2643 tablet_replica.cc:439] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } } }
I20260620 14:31:28.060967 2665 mvcc.cc:205] Tried to move back new op lower bound from 7298932277477806080 to 7298932277435543552. Current Snapshot: MvccSnapshot[applied={T|T < 7298932277477806080}]
I20260620 14:31:28.061026 2643 tablet_replica.cc:442] T 94612692d35b4c62b41572e484863929 P 909bd0ae228244319ac0958eae66620f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:28.064381 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.062142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.98.3:42805)
0620 14:31:28.062186 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.062187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.062195 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:28.062272 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.062273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.062285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.062362 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.062468 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.062901 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.063622 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.063840 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.063922 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.064084 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.064178 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.064264 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.064286 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.064308 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:28.069036 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.066693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44473 (local address 127.0.98.1:33819)
0620 14:31:28.066753 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.066755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.066764 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:28.066783 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.066784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.066810 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.066855 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.066987 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.067388 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.068462 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.068625 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.068671 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.068863 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.068886 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.068921 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.068976 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.068991 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:31:28.069082 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.066620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.98.3:44473)
0620 14:31:28.066688 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.066697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.066730 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.066874 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.066875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.066876 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:28.066931 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.066932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.067467 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.067468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.068386 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.068388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.068690 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.068691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.068756 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.068771 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.068922 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.068924 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.068974 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.069012 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:31:28.071442 2430 raft_consensus.cc:1261] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Refusing update from remote peer 909bd0ae228244319ac0958eae66620f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.071475 2290 raft_consensus.cc:1261] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Refusing update from remote peer 909bd0ae228244319ac0958eae66620f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.071695 2643 consensus_queue.cc:1035] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:28.071811 2646 consensus_queue.cc:1035] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:28.088177 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.077500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54889 (local address 127.0.98.3:42805)
0620 14:31:28.077536 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.077538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.085619 (+ 8081us) server_negotiation.cc:407] Connection header received
0620 14:31:28.085683 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.085684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.085696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.085751 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.085945 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.086420 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.087366 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.087676 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.087743 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.087955 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.087983 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.088026 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.088100 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.088117 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:31:28.088258 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.077459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42805 (local address 127.0.98.1:54889)
0620 14:31:28.085547 (+ 8088us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.085558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.085631 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.085766 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.085767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.085768 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:28.085852 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.085854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.086502 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.086503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.087179 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.087182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.087792 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.087793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.087862 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.087880 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.088043 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.088045 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.088099 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.088143 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8049,"thread_start_us":70,"threads_started":1}
I20260620 14:31:28.097532 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:28.097659 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:28.098358 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Sending DeleteTablet for 3 replicas of tablet 6b8589dc2bd5497a856512ab6699d0d7
I20260620 14:31:28.098712 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260620 14:31:28.098722 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58310
I20260620 14:31:28.099040 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b8589dc2bd5497a856512ab6699d0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260620 14:31:28.099124 2683 tablet_replica.cc:330] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: stopping tablet replica
I20260620 14:31:28.099254 2685 tablet_replica.cc:330] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: stopping tablet replica
I20260620 14:31:28.099252 2684 tablet_replica.cc:330] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: stopping tablet replica
I20260620 14:31:28.099351 2684 raft_consensus.cc:2229] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:28.099393 2685 raft_consensus.cc:2229] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:28.099479 2685 raft_consensus.cc:2258] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:28.099418 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:28.099548 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:28.099720 2684 raft_consensus.cc:2258] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:28.099815 2683 raft_consensus.cc:2229] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:28.099934 2683 raft_consensus.cc:2258] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:28.100839 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 813c0e6960874246ba6d131532493cde (DEFAULT_TABLE table=test-table [id=a69c74d089d847e48e9813f1b46ebeca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:28.100850 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 813c0e6960874246ba6d131532493cde (DEFAULT_TABLE table=test-table [id=a69c74d089d847e48e9813f1b46ebeca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:28.100991 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813c0e6960874246ba6d131532493cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:28.100998 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813c0e6960874246ba6d131532493cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:28.102315 2638 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Bootstrap starting.
I20260620 14:31:28.102357 2636 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Bootstrap starting.
I20260620 14:31:28.102614 2683 ts_tablet_manager.cc:1905] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:28.102737 2636 tablet_bootstrap.cc:654] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:28.102739 2638 tablet_bootstrap.cc:654] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:28.103214 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 813c0e6960874246ba6d131532493cde (DEFAULT_TABLE table=test-table [id=a69c74d089d847e48e9813f1b46ebeca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:28.103329 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813c0e6960874246ba6d131532493cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:28.104310 2683 ts_tablet_manager.cc:1918] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:28.104516 2683 log.cc:1196] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/6b8589dc2bd5497a856512ab6699d0d7
I20260620 14:31:28.104580 2639 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Bootstrap starting.
I20260620 14:31:28.104794 2685 ts_tablet_manager.cc:1905] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:28.104854 2683 ts_tablet_manager.cc:1939] T 6b8589dc2bd5497a856512ab6699d0d7 P 9851a3bee3bc428486822f321d585606: Deleting consensus metadata
I20260620 14:31:28.105000 2639 tablet_bootstrap.cc:654] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:28.105261 2117 catalog_manager.cc:4841] TS 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285): tablet 6b8589dc2bd5497a856512ab6699d0d7 (table test-table [id=c510ba4308cc416db359baaa64c0c0f2]) successfully deleted
I20260620 14:31:28.105903 2639 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: No bootstrap required, opened a new log
I20260620 14:31:28.105973 2639 ts_tablet_manager.cc:1397] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:28.106109 2639 raft_consensus.cc:348] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.106163 2639 raft_consensus.cc:374] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:28.106177 2639 raft_consensus.cc:729] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Initialized, Role: FOLLOWER
I20260620 14:31:28.106227 2685 ts_tablet_manager.cc:1918] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:28.106307 2685 log.cc:1196] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/6b8589dc2bd5497a856512ab6699d0d7
I20260620 14:31:28.106233 2639 consensus_queue.cc:260] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.106498 2639 ts_tablet_manager.cc:1428] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:28.106561 2685 ts_tablet_manager.cc:1939] T 6b8589dc2bd5497a856512ab6699d0d7 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting consensus metadata
I20260620 14:31:28.107203 2684 ts_tablet_manager.cc:1905] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:28.108472 2684 ts_tablet_manager.cc:1918] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:28.108553 2684 log.cc:1196] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/6b8589dc2bd5497a856512ab6699d0d7
I20260620 14:31:28.108791 2684 ts_tablet_manager.cc:1939] T 6b8589dc2bd5497a856512ab6699d0d7 P 909bd0ae228244319ac0958eae66620f: Deleting consensus metadata
I20260620 14:31:28.109174 2118 catalog_manager.cc:4841] TS 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805): tablet 6b8589dc2bd5497a856512ab6699d0d7 (table test-table [id=c510ba4308cc416db359baaa64c0c0f2]) successfully deleted
I20260620 14:31:28.109582 2119 catalog_manager.cc:4841] TS 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819): tablet 6b8589dc2bd5497a856512ab6699d0d7 (table test-table [id=c510ba4308cc416db359baaa64c0c0f2]) successfully deleted
I20260620 14:31:28.113309 2636 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: No bootstrap required, opened a new log
I20260620 14:31:28.113363 2636 ts_tablet_manager.cc:1397] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 14:31:28.113504 2638 tablet_bootstrap.cc:492] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: No bootstrap required, opened a new log
I20260620 14:31:28.113540 2638 ts_tablet_manager.cc:1397] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 14:31:28.113540 2636 raft_consensus.cc:348] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.113617 2636 raft_consensus.cc:374] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:28.113641 2636 raft_consensus.cc:729] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851a3bee3bc428486822f321d585606, State: Initialized, Role: FOLLOWER
I20260620 14:31:28.113688 2636 consensus_queue.cc:260] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.113688 2638 raft_consensus.cc:348] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.113749 2638 raft_consensus.cc:374] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:28.113770 2638 raft_consensus.cc:729] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Initialized, Role: FOLLOWER
I20260620 14:31:28.113780 2636 ts_tablet_manager.cc:1428] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:28.113811 2638 consensus_queue.cc:260] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.113895 2638 ts_tablet_manager.cc:1428] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:28.138477 2671 raft_consensus.cc:482] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:28.138590 2671 raft_consensus.cc:504] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.138753 2671 leader_election.cc:288] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:28.138964 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813c0e6960874246ba6d131532493cde" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606" is_pre_election: true
I20260620 14:31:28.139060 2430 raft_consensus.cc:2454] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:28.139026 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813c0e6960874246ba6d131532493cde" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f" is_pre_election: true
I20260620 14:31:28.139106 2570 raft_consensus.cc:2454] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:28.139218 2221 leader_election.cc:302] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 9851a3bee3bc428486822f321d585606; no voters:
I20260620 14:31:28.139436 2671 raft_consensus.cc:2781] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:28.139478 2671 raft_consensus.cc:482] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:28.139506 2671 raft_consensus.cc:3037] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.140297 2671 raft_consensus.cc:504] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.140442 2671 leader_election.cc:288] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Requested vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:28.140609 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813c0e6960874246ba6d131532493cde" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606"
I20260620 14:31:28.140638 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813c0e6960874246ba6d131532493cde" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f"
I20260620 14:31:28.140671 2430 raft_consensus.cc:3037] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.140694 2570 raft_consensus.cc:3037] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:28.141373 2430 raft_consensus.cc:2454] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:28.141373 2570 raft_consensus.cc:2454] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:28.141542 2221 leader_election.cc:302] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 9851a3bee3bc428486822f321d585606; no voters:
I20260620 14:31:28.141651 2671 raft_consensus.cc:2781] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:28.141700 2671 raft_consensus.cc:686] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Running, Role: LEADER
I20260620 14:31:28.141752 2671 consensus_queue.cc:237] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:28.142300 2131 catalog_manager.cc:5495] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:28.151883 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.149553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.98.1:33819)
0620 14:31:28.149591 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.149593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.149629 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:31:28.149713 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.149714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.149724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.149779 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.149886 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.150413 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.151199 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.151400 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.151473 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.151619 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.151652 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.151721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.151781 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.151800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:31:28.155872 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.153887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49561 (local address 127.0.98.1:33819)
0620 14:31:28.153922 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:28.153924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:28.153931 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:28.153982 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:28.153983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:28.153991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:28.154041 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:28.154128 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.154546 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.155244 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.155439 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.155518 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.155629 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.155656 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.155706 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:28.155765 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:31:28.155796 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":14}
I20260620 14:31:28.155934 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:28.153843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.98.1:49561)
0620 14:31:28.153876 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:28.153887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:28.153927 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:28.154044 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:28.154044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:28.154045 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:28.154076 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.154077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.154645 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.154646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.155166 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:28.155167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:28.155517 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:28.155517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:28.155564 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:28.155568 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:28.155710 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:28.155711 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:31:28.155761 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:31:28.155812 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:31:28.158120 2430 raft_consensus.cc:1261] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.158120 2570 raft_consensus.cc:1261] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:28.158315 2644 consensus_queue.cc:1035] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:28.158418 2644 consensus_queue.cc:1035] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:31:28.162833 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:31:29.682961 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.683185 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.683970 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Sending DeleteTablet for 3 replicas of tablet 813c0e6960874246ba6d131532493cde
I20260620 14:31:29.684317 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 813c0e6960874246ba6d131532493cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260620 14:31:29.684363 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 813c0e6960874246ba6d131532493cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58310
I20260620 14:31:29.684541 2713 tablet_replica.cc:330] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: stopping tablet replica
I20260620 14:31:29.684587 2714 tablet_replica.cc:330] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: stopping tablet replica
I20260620 14:31:29.684604 2713 raft_consensus.cc:2229] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:29.684691 2714 raft_consensus.cc:2229] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.684702 2713 raft_consensus.cc:2258] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.684736 2714 raft_consensus.cc:2258] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.684955 2714 ts_tablet_manager.cc:1905] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.685654 2713 ts_tablet_manager.cc:1905] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.686697 2714 ts_tablet_manager.cc:1918] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:31:29.686787 2714 log.cc:1196] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/813c0e6960874246ba6d131532493cde
I20260620 14:31:29.686836 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 813c0e6960874246ba6d131532493cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260620 14:31:29.687031 2713 ts_tablet_manager.cc:1918] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:31:29.687063 2714 ts_tablet_manager.cc:1939] T 813c0e6960874246ba6d131532493cde P 9851a3bee3bc428486822f321d585606: Deleting consensus metadata
I20260620 14:31:29.687109 2713 log.cc:1196] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/813c0e6960874246ba6d131532493cde
I20260620 14:31:29.687376 2713 ts_tablet_manager.cc:1939] T 813c0e6960874246ba6d131532493cde P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting consensus metadata
I20260620 14:31:29.687415 2117 catalog_manager.cc:4841] TS 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285): tablet 813c0e6960874246ba6d131532493cde (table test-table [id=a69c74d089d847e48e9813f1b46ebeca]) successfully deleted
I20260620 14:31:29.687716 2119 catalog_manager.cc:4841] TS 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819): tablet 813c0e6960874246ba6d131532493cde (table test-table [id=a69c74d089d847e48e9813f1b46ebeca]) successfully deleted
I20260620 14:31:29.688377 2716 tablet_replica.cc:330] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: stopping tablet replica
I20260620 14:31:29.688439 2716 raft_consensus.cc:2229] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.688491 2716 raft_consensus.cc:2258] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.688697 2716 ts_tablet_manager.cc:1905] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.690042 2716 ts_tablet_manager.cc:1918] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:31:29.690089 2716 log.cc:1196] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/813c0e6960874246ba6d131532493cde
I20260620 14:31:29.690282 2716 ts_tablet_manager.cc:1939] T 813c0e6960874246ba6d131532493cde P 909bd0ae228244319ac0958eae66620f: Deleting consensus metadata
I20260620 14:31:29.690570 2118 catalog_manager.cc:4841] TS 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805): tablet 813c0e6960874246ba6d131532493cde (table test-table [id=a69c74d089d847e48e9813f1b46ebeca]) successfully deleted
Jun 20 14:31:29 dist-test-slave-tc2g krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:29 dist-test-slave-tc2g krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:29.708290 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.701508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.98.62:46445)
0620 14:31:29.701678 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.701680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.701689 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:29.701744 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.701744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.701755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.701813 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.702377 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.702861 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.703566 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.703745 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.703794 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.706043 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.706045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.706045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.706051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.707304 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.707641 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.707642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.707642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.707689 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.707854 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.707855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.707855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.707963 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.708037 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.708124 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.708182 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260620 14:31:29.709923 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:29.710067 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:29.711339 2406 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd9d9ae730440398255d3034b03241 (DEFAULT_TABLE table=test-table [id=f99508c5fdf64715a7a54421a912deac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.711339 2269 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd9d9ae730440398255d3034b03241 (DEFAULT_TABLE table=test-table [id=f99508c5fdf64715a7a54421a912deac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.711442 2550 tablet_service.cc:1467] Processing CreateTablet for tablet a2fd9d9ae730440398255d3034b03241 (DEFAULT_TABLE table=test-table [id=f99508c5fdf64715a7a54421a912deac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.711475 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd9d9ae730440398255d3034b03241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.711476 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd9d9ae730440398255d3034b03241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.711565 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2fd9d9ae730440398255d3034b03241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.712651 2724 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Bootstrap starting.
I20260620 14:31:29.712849 2725 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Bootstrap starting.
I20260620 14:31:29.713074 2726 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Bootstrap starting.
I20260620 14:31:29.713213 2724 tablet_bootstrap.cc:654] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.713330 2725 tablet_bootstrap.cc:654] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.713549 2726 tablet_bootstrap.cc:654] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.713874 2724 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: No bootstrap required, opened a new log
I20260620 14:31:29.713950 2724 ts_tablet_manager.cc:1397] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:29.713994 2725 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: No bootstrap required, opened a new log
I20260620 14:31:29.714046 2725 ts_tablet_manager.cc:1397] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:29.714076 2724 raft_consensus.cc:348] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.714202 2724 raft_consensus.cc:374] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.714234 2724 raft_consensus.cc:729] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.714201 2725 raft_consensus.cc:348] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.714291 2725 raft_consensus.cc:374] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.714287 2724 consensus_queue.cc:260] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.714318 2725 raft_consensus.cc:729] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851a3bee3bc428486822f321d585606, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.714360 2725 consensus_queue.cc:260] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.714462 2724 ts_tablet_manager.cc:1428] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:29.714517 2725 ts_tablet_manager.cc:1428] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:29.714783 2726 tablet_bootstrap.cc:492] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: No bootstrap required, opened a new log
I20260620 14:31:29.714850 2726 ts_tablet_manager.cc:1397] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:31:29.714964 2726 raft_consensus.cc:348] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.715037 2726 raft_consensus.cc:374] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.715062 2726 raft_consensus.cc:729] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.715101 2726 consensus_queue.cc:260] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.715194 2726 ts_tablet_manager.cc:1428] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.836896 2731 raft_consensus.cc:482] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:29.837008 2731 raft_consensus.cc:504] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.837383 2731 leader_election.cc:288] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:29.840587 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.837602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42805 (local address 127.0.98.2:50585)
0620 14:31:29.837819 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.837829 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.837924 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.838095 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.838096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.838097 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.838154 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.838157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.838840 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.838841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.839744 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.839747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.840297 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.840298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.840369 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.840373 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.840460 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.840500 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":131,"threads_started":1}
I20260620 14:31:29.840576 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.837675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50585 (local address 127.0.98.3:42805)
0620 14:31:29.837823 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.837825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.837907 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:31:29.837992 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.837992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.838003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.838091 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.838241 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.838755 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.839835 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.840212 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.840297 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.840309 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.840472 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.840510 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260620 14:31:29.840736 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.837609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.98.2:49253)
0620 14:31:29.837818 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.837830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.837924 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.838153 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.838154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.838156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.838489 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.838490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.839201 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.839202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.839919 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.839921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.840480 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.840481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.840551 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.840554 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.840622 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.840659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":131,"threads_started":1}
I20260620 14:31:29.840991 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.837647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49253 (local address 127.0.98.1:33819)
0620 14:31:29.837890 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.837892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.837902 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:29.837998 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.837999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.838014 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.838149 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.838569 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.839092 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.840024 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.840381 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.840477 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.840488 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.840627 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.840914 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260620 14:31:29.840999 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd9d9ae730440398255d3034b03241" candidate_uuid: "9851a3bee3bc428486822f321d585606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f" is_pre_election: true
I20260620 14:31:29.841097 2569 raft_consensus.cc:2454] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9851a3bee3bc428486822f321d585606 in term 0.
I20260620 14:31:29.841164 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd9d9ae730440398255d3034b03241" candidate_uuid: "9851a3bee3bc428486822f321d585606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" is_pre_election: true
I20260620 14:31:29.841275 2290 raft_consensus.cc:2454] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9851a3bee3bc428486822f321d585606 in term 0.
I20260620 14:31:29.841311 2362 leader_election.cc:302] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909bd0ae228244319ac0958eae66620f, 9851a3bee3bc428486822f321d585606; no voters:
I20260620 14:31:29.841452 2731 raft_consensus.cc:2781] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:29.841511 2731 raft_consensus.cc:482] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:29.841535 2731 raft_consensus.cc:3037] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.842185 2731 raft_consensus.cc:504] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.842326 2731 leader_election.cc:288] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [CANDIDATE]: Term 1 election: Requested vote from peers 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:29.842453 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd9d9ae730440398255d3034b03241" candidate_uuid: "9851a3bee3bc428486822f321d585606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f"
I20260620 14:31:29.842463 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fd9d9ae730440398255d3034b03241" candidate_uuid: "9851a3bee3bc428486822f321d585606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7f3fd6a6634fc0b5760701b65c6276"
I20260620 14:31:29.842533 2569 raft_consensus.cc:3037] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.842533 2290 raft_consensus.cc:3037] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.843271 2569 raft_consensus.cc:2454] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9851a3bee3bc428486822f321d585606 in term 1.
I20260620 14:31:29.843271 2290 raft_consensus.cc:2454] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9851a3bee3bc428486822f321d585606 in term 1.
I20260620 14:31:29.843444 2362 leader_election.cc:302] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909bd0ae228244319ac0958eae66620f, 9851a3bee3bc428486822f321d585606; no voters:
I20260620 14:31:29.843528 2731 raft_consensus.cc:2781] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:29.843672 2731 raft_consensus.cc:686] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 1 LEADER]: Becoming Leader. State: Replica: 9851a3bee3bc428486822f321d585606, State: Running, Role: LEADER
I20260620 14:31:29.843748 2731 consensus_queue.cc:237] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.844411 2133 catalog_manager.cc:5495] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9851a3bee3bc428486822f321d585606 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9851a3bee3bc428486822f321d585606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:29.861254 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.859015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.0.98.2:33285)
0620 14:31:29.859151 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.859152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.859159 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:29.859191 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.859192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.859209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.859278 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:29.859389 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.859860 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.860553 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.860738 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.860786 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.860934 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.861036 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.861090 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.861122 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.861138 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260620 14:31:29.865896 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.863643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37123 (local address 127.0.98.1:33819)
0620 14:31:29.863706 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.863708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.863718 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:29.863773 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.863774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.863785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.863839 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:29.863998 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.864467 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.865202 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.865402 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.865556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.865710 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.865737 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.865775 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.865824 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.865842 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:31:29.865929 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.863494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.98.2:37123)
0620 14:31:29.863633 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.863646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.863718 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.863842 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.863842 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.863844 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:29.863907 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.863908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.864545 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.864546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.865117 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.865119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.865562 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.865563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.865624 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.865636 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.865786 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.865788 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.865822 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.865855 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 14:31:29.868072 2290 raft_consensus.cc:1261] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Refusing update from remote peer 9851a3bee3bc428486822f321d585606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:29.868137 2569 raft_consensus.cc:1261] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Refusing update from remote peer 9851a3bee3bc428486822f321d585606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:29.868387 2736 consensus_queue.cc:1035] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:29.868546 2731 consensus_queue.cc:1035] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:29.870229 2693 mvcc.cc:205] Tried to move back new op lower bound from 7298932284897820672 to 7298932284800475136. Current Snapshot: MvccSnapshot[applied={T|T < 7298932284897820672}]
I20260620 14:31:29.875548 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.873248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55777 (local address 127.0.98.2:33285)
0620 14:31:29.873285 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.873287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.873395 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:31:29.873494 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.873495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.873506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.873554 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:29.873662 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.874125 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.874867 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.875073 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.875148 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.875302 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.875333 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.875382 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.875466 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.875484 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:31:29.875583 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.873210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33285 (local address 127.0.98.1:55777)
0620 14:31:29.873324 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.873334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.873408 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.873556 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.873556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.873558 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:29.873605 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.873606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.874205 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.874206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.874770 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.874771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.875147 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.875148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.875214 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.875219 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.875409 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.875411 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.875461 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.875506 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260620 14:31:29.881983 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.882084 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.882715 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Sending DeleteTablet for 3 replicas of tablet a2fd9d9ae730440398255d3034b03241
I20260620 14:31:29.882961 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fd9d9ae730440398255d3034b03241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260620 14:31:29.882992 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fd9d9ae730440398255d3034b03241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260620 14:31:29.883046 2713 tablet_replica.cc:330] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: stopping tablet replica
I20260620 14:31:29.883080 2716 tablet_replica.cc:330] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: stopping tablet replica
I20260620 14:31:29.883088 2713 raft_consensus.cc:2229] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.883117 2713 raft_consensus.cc:2258] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.883131 2716 raft_consensus.cc:2229] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.883163 2716 raft_consensus.cc:2258] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.883419 2713 ts_tablet_manager.cc:1905] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.883576 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:29.883805 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:29.884884 2269 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a37160374016a83d542670406ad1 (DEFAULT_TABLE table=test-table [id=03bed8b9321c4a739ebe18d11190fc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.885048 2550 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a37160374016a83d542670406ad1 (DEFAULT_TABLE table=test-table [id=03bed8b9321c4a739ebe18d11190fc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.885136 2410 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a37160374016a83d542670406ad1 (DEFAULT_TABLE table=test-table [id=03bed8b9321c4a739ebe18d11190fc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.885176 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a37160374016a83d542670406ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.885270 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a37160374016a83d542670406ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.886396 2724 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Bootstrap starting.
I20260620 14:31:29.886600 2725 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Bootstrap starting.
I20260620 14:31:29.886713 2724 tablet_bootstrap.cc:654] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.886966 2725 tablet_bootstrap.cc:654] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.887127 2716 ts_tablet_manager.cc:1905] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.887557 2725 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: No bootstrap required, opened a new log
I20260620 14:31:29.887622 2725 ts_tablet_manager.cc:1397] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:29.887797 2725 raft_consensus.cc:348] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.887897 2725 raft_consensus.cc:374] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.887926 2725 raft_consensus.cc:729] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9851a3bee3bc428486822f321d585606, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.887961 2725 consensus_queue.cc:260] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.888079 2725 ts_tablet_manager.cc:1428] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:29.888440 2716 ts_tablet_manager.cc:1918] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:29.888496 2716 log.cc:1196] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/a2fd9d9ae730440398255d3034b03241
I20260620 14:31:29.888706 2716 ts_tablet_manager.cc:1939] T a2fd9d9ae730440398255d3034b03241 P 909bd0ae228244319ac0958eae66620f: Deleting consensus metadata
I20260620 14:31:29.888834 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fd9d9ae730440398255d3034b03241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58310
I20260620 14:31:29.888921 2714 tablet_replica.cc:330] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: stopping tablet replica
I20260620 14:31:29.888967 2714 raft_consensus.cc:2229] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:29.889047 2714 raft_consensus.cc:2258] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.889233 2118 catalog_manager.cc:4841] TS 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805): tablet a2fd9d9ae730440398255d3034b03241 (table test-table [id=f99508c5fdf64715a7a54421a912deac]) successfully deleted
I20260620 14:31:29.889392 2724 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: No bootstrap required, opened a new log
I20260620 14:31:29.889477 2724 ts_tablet_manager.cc:1397] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:29.889533 2713 ts_tablet_manager.cc:1918] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:29.889590 2713 log.cc:1196] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/a2fd9d9ae730440398255d3034b03241
I20260620 14:31:29.889580 2724 raft_consensus.cc:348] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.889645 2724 raft_consensus.cc:374] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.889670 2724 raft_consensus.cc:729] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909bd0ae228244319ac0958eae66620f, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.889719 2724 consensus_queue.cc:260] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.889812 2713 ts_tablet_manager.cc:1939] T a2fd9d9ae730440398255d3034b03241 P 3a7f3fd6a6634fc0b5760701b65c6276: Deleting consensus metadata
I20260620 14:31:29.889820 2724 ts_tablet_manager.cc:1428] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.889946 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a37160374016a83d542670406ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.890210 2119 catalog_manager.cc:4841] TS 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819): tablet a2fd9d9ae730440398255d3034b03241 (table test-table [id=f99508c5fdf64715a7a54421a912deac]) successfully deleted
I20260620 14:31:29.890436 2714 ts_tablet_manager.cc:1905] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.891091 2726 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276: Bootstrap starting.
I20260620 14:31:29.891419 2726 tablet_bootstrap.cc:654] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.891747 2714 ts_tablet_manager.cc:1918] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:29.891800 2714 log.cc:1196] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/a2fd9d9ae730440398255d3034b03241
I20260620 14:31:29.891924 2726 tablet_bootstrap.cc:492] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276: No bootstrap required, opened a new log
I20260620 14:31:29.891956 2726 ts_tablet_manager.cc:1397] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:29.892004 2714 ts_tablet_manager.cc:1939] T a2fd9d9ae730440398255d3034b03241 P 9851a3bee3bc428486822f321d585606: Deleting consensus metadata
I20260620 14:31:29.892027 2726 raft_consensus.cc:348] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.892072 2726 raft_consensus.cc:374] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.892086 2726 raft_consensus.cc:729] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.892140 2726 consensus_queue.cc:260] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.892232 2726 ts_tablet_manager.cc:1428] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.892469 2117 catalog_manager.cc:4841] TS 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285): tablet a2fd9d9ae730440398255d3034b03241 (table test-table [id=f99508c5fdf64715a7a54421a912deac]) successfully deleted
I20260620 14:31:29.984609 2644 raft_consensus.cc:482] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:29.984724 2644 raft_consensus.cc:504] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.984884 2644 leader_election.cc:288] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:29.985052 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a37160374016a83d542670406ad1" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606" is_pre_election: true
I20260620 14:31:29.985059 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a37160374016a83d542670406ad1" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f" is_pre_election: true
I20260620 14:31:29.985164 2430 raft_consensus.cc:2454] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:29.985179 2569 raft_consensus.cc:2454] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 0.
I20260620 14:31:29.985338 2222 leader_election.cc:302] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 909bd0ae228244319ac0958eae66620f; no voters:
I20260620 14:31:29.985496 2644 raft_consensus.cc:2781] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:29.985541 2644 raft_consensus.cc:482] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:29.985561 2644 raft_consensus.cc:3037] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.986235 2644 raft_consensus.cc:504] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.986382 2644 leader_election.cc:288] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Requested vote from peers 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285), 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805)
I20260620 14:31:29.986567 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a37160374016a83d542670406ad1" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9851a3bee3bc428486822f321d585606"
I20260620 14:31:29.986577 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a37160374016a83d542670406ad1" candidate_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909bd0ae228244319ac0958eae66620f"
I20260620 14:31:29.986649 2430 raft_consensus.cc:3037] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.986668 2569 raft_consensus.cc:3037] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.987231 2569 raft_consensus.cc:2454] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:29.987382 2430 raft_consensus.cc:2454] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a7f3fd6a6634fc0b5760701b65c6276 in term 1.
I20260620 14:31:29.987396 2222 leader_election.cc:302] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a7f3fd6a6634fc0b5760701b65c6276, 909bd0ae228244319ac0958eae66620f; no voters:
I20260620 14:31:29.987507 2644 raft_consensus.cc:2781] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:29.987550 2644 raft_consensus.cc:686] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7f3fd6a6634fc0b5760701b65c6276, State: Running, Role: LEADER
I20260620 14:31:29.987600 2644 consensus_queue.cc:237] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } }
I20260620 14:31:29.988162 2131 catalog_manager.cc:5495] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a7f3fd6a6634fc0b5760701b65c6276" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:30.007793 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:30.005570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.0.98.1:33819)
0620 14:31:30.005616 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:30.005618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:30.005628 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:30.005688 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:30.005688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:30.005698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:30.005744 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:30.005842 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:30.006299 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:30.007074 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:30.007279 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:30.007354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:30.007496 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:30.007532 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:30.007617 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:30.007669 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:31:30.007690 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:30.009953 2569 raft_consensus.cc:1261] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:30.010108 2430 raft_consensus.cc:1261] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Refusing update from remote peer 3a7f3fd6a6634fc0b5760701b65c6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:30.010138 2695 consensus_queue.cc:1035] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909bd0ae228244319ac0958eae66620f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:30.010421 2695 consensus_queue.cc:1035] T b0d3a37160374016a83d542670406ad1 P 3a7f3fd6a6634fc0b5760701b65c6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9851a3bee3bc428486822f321d585606" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:31:30.015239 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:31:31.531906 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:31.532015 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:31.532816 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e765455ba484ead86f6bb1c0cfb0e: Sending DeleteTablet for 3 replicas of tablet b0d3a37160374016a83d542670406ad1
I20260620 14:31:31.533077 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a37160374016a83d542670406ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260620 14:31:31.533095 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a37160374016a83d542670406ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58310
I20260620 14:31:31.533118 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a37160374016a83d542670406ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55248
I20260620 14:31:31.533262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2205
I20260620 14:31:31.533324 2777 tablet_replica.cc:330] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: stopping tablet replica
I20260620 14:31:31.535182 2776 tablet_replica.cc:330] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: stopping tablet replica
I20260620 14:31:31.535244 2776 raft_consensus.cc:2229] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:31.535285 2776 raft_consensus.cc:2258] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:31.535422 2777 raft_consensus.cc:2229] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:31.535487 2777 raft_consensus.cc:2258] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:31.535735 2777 ts_tablet_manager.cc:1905] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:31.537503 2776 ts_tablet_manager.cc:1905] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:31.537796 2777 ts_tablet_manager.cc:1918] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:31:31.537860 2777 log.cc:1196] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/b0d3a37160374016a83d542670406ad1
I20260620 14:31:31.538127 2777 ts_tablet_manager.cc:1939] T b0d3a37160374016a83d542670406ad1 P 909bd0ae228244319ac0958eae66620f: Deleting consensus metadata
I20260620 14:31:31.538445 2118 catalog_manager.cc:4841] TS 909bd0ae228244319ac0958eae66620f (127.0.98.3:42805): tablet b0d3a37160374016a83d542670406ad1 (table test-table [id=03bed8b9321c4a739ebe18d11190fc65]) successfully deleted
I20260620 14:31:31.539022 2776 ts_tablet_manager.cc:1918] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:31:31.539094 2776 log.cc:1196] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/b0d3a37160374016a83d542670406ad1
I20260620 14:31:31.539288 2776 ts_tablet_manager.cc:1939] T b0d3a37160374016a83d542670406ad1 P 9851a3bee3bc428486822f321d585606: Deleting consensus metadata
I20260620 14:31:31.539713 2117 catalog_manager.cc:4841] TS 9851a3bee3bc428486822f321d585606 (127.0.98.2:33285): tablet b0d3a37160374016a83d542670406ad1 (table test-table [id=03bed8b9321c4a739ebe18d11190fc65]) successfully deleted
W20260620 14:31:31.541280 2119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33819 (error 108)
I20260620 14:31:31.541473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2345
I20260620 14:31:31.542470 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:31.542266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33819 (local address 127.0.0.1:53636)
0620 14:31:31.542386 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:31.542436 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
W20260620 14:31:31.542577 2119 catalog_manager.cc:4584] TS 3a7f3fd6a6634fc0b5760701b65c6276 (127.0.98.1:33819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0d3a37160374016a83d542670406ad1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33819: connect: Connection refused (error 111)
I20260620 14:31:31.548060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2485
I20260620 14:31:31.553362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2101
2026-06-20T14:31:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5557 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:31 dist-test-slave-tc2g krb5kdc[2782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:31 dist-test-slave-tc2g krb5kdc[2782](info): set up 2 sockets
Jun 20 14:31:31 dist-test-slave-tc2g krb5kdc[2782](info): commencing operation
krb5kdc: starting...
W20260620 14:31:32.597240 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:32 dist-test-slave-tc2g krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:32.711913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:32.790728 2795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:32.790977 2795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:32.791014 2795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:32.792155 2795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:32.792225 2795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:32.792248 2795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:32.792270 2795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:32.792297 2795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:32.793835 2795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:32.794123 2795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:32.794342 2795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:32.799026 2800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:32.799117 2795 server_base.cc:1034] running on GCE node
W20260620 14:31:32.799039 2801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:32.799026 2803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:32.799474 2795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:32.799772 2795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:32.800942 2795 hybrid_clock.cc:648] HybridClock initialized: now 1781965892800934 us; error 36 us; skew 500 ppm
Jun 20 14:31:32 dist-test-slave-tc2g krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:32.803447 2795 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:32.803920 2795 webserver.cc:460] Webserver started at http://127.0.98.62:41947/ using document root <none> and password file <none>
I20260620 14:31:32.804163 2795 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:32.804229 2795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:32.804381 2795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:32.805483 2795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "4147d9a532b34f319f92d36dc60891f8"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "5c54a6b54c83a9d085a4cad35f023af1"
server_key_iv: "39bfff9bb771fa2f10e2695fd435b5ae"
server_key_version: "encryptionkey@0"
I20260620 14:31:32.805891 2795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "4147d9a532b34f319f92d36dc60891f8"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "5c54a6b54c83a9d085a4cad35f023af1"
server_key_iv: "39bfff9bb771fa2f10e2695fd435b5ae"
server_key_version: "encryptionkey@0"
I20260620 14:31:32.807219 2795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:32.807920 2810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:32.808099 2795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:32.808194 2795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "4147d9a532b34f319f92d36dc60891f8"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "5c54a6b54c83a9d085a4cad35f023af1"
server_key_iv: "39bfff9bb771fa2f10e2695fd435b5ae"
server_key_version: "encryptionkey@0"
I20260620 14:31:32.808274 2795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:32.821166 2795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:32.823493 2795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:32.823647 2795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:32.827661 2795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46463
I20260620 14:31:32.827703 2862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46463 every 8 connection(s)
I20260620 14:31:32.828186 2795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:32.828857 2863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:32.831143 2863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8: Bootstrap starting.
I20260620 14:31:32.831718 2863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:32.832003 2863 log.cc:824] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:32.832690 2863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8: No bootstrap required, opened a new log
I20260620 14:31:32.834280 2863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260620 14:31:32.834414 2863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:32.834435 2863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4147d9a532b34f319f92d36dc60891f8, State: Initialized, Role: FOLLOWER
I20260620 14:31:32.834585 2863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260620 14:31:32.834656 2863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:32.834695 2863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:32.834744 2863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:32.835599 2863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260620 14:31:32.835696 2863 leader_election.cc:302] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4147d9a532b34f319f92d36dc60891f8; no voters:
I20260620 14:31:32.835863 2863 leader_election.cc:288] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:32.835909 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:32.836100 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4147d9a532b34f319f92d36dc60891f8, State: Running, Role: LEADER
I20260620 14:31:32.836138 2863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:32.836222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2795
I20260620 14:31:32.836305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:32.836282 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260620 14:31:32.836504 392 external_mini_cluster.cc:1428] Setting key 767e8c9f66a983faaf8ee0f9752810db
I20260620 14:31:32.836653 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4147d9a532b34f319f92d36dc60891f8. Latest consensus state: current_term: 1 leader_uuid: "4147d9a532b34f319f92d36dc60891f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } } }
I20260620 14:31:32.836758 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:32.836679 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4147d9a532b34f319f92d36dc60891f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147d9a532b34f319f92d36dc60891f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } } }
I20260620 14:31:32.837581 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:32.837610 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:32.838145 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:32.839674 2881 catalog_manager.cc:1347] Generated new cluster ID: 4fab7b4fc6994dc5944097e7166ad722
I20260620 14:31:32.839726 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:32 dist-test-slave-tc2g krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:32.847327 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:32.837173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.0.98.62:46463)
0620 14:31:32.837689 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:32.837699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:32.837734 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:31:32.837909 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:32.837917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:32.837981 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:32.838147 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:32.838765 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:32.839903 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:32.840714 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:32.841130 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:32.841258 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:32.843566 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:32.843570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:32.843570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:32.843582 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:32.845785 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:32.846160 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:32.846163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:32.846163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:32.846252 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:32.846457 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:32.846458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:32.846458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:32.846701 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:32.846872 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:32.846966 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:31:32.847111 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":69,"threads_started":1}
I20260620 14:31:32.859164 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:32.859714 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:32.862443 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4147d9a532b34f319f92d36dc60891f8: Generated new TSK 0
I20260620 14:31:32.862643 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:32.894608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:32.985749 2889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:32.985965 2889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:32.986008 2889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:32.987196 2889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:32.987258 2889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:32.987303 2889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:32.988838 2889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:32.989113 2889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:32.989333 2889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:32.994151 2894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:32.994158 2897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:32.994159 2895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:32.994570 2889 server_base.cc:1034] running on GCE node
I20260620 14:31:32.994808 2889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:32.995059 2889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:32.996224 2889 hybrid_clock.cc:648] HybridClock initialized: now 1781965892996216 us; error 37 us; skew 500 ppm
Jun 20 14:31:32 dist-test-slave-tc2g krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:32.998759 2889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:32.999197 2889 webserver.cc:460] Webserver started at http://127.0.98.1:34277/ using document root <none> and password file <none>
I20260620 14:31:32.999429 2889 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:32.999495 2889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:32.999637 2889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:33.000635 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5539d9aaea94c879db4fdca757261c7"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "642533d08b3b297ea0a9acb0f1bf7893"
server_key_iv: "4013a1ac34be13a19bf9b93cec241950"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.000984 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5539d9aaea94c879db4fdca757261c7"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "642533d08b3b297ea0a9acb0f1bf7893"
server_key_iv: "4013a1ac34be13a19bf9b93cec241950"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.002377 2889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:33.003079 2904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.003266 2889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:33.003361 2889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "f5539d9aaea94c879db4fdca757261c7"
format_stamp: "Formatted at 2026-06-20 14:31:32 on dist-test-slave-tc2g"
server_key: "642533d08b3b297ea0a9acb0f1bf7893"
server_key_iv: "4013a1ac34be13a19bf9b93cec241950"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.003434 2889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:33.019944 2889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:33.022339 2889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:33.022500 2889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:33.022748 2889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:33.023036 2889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:33.023084 2889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.023116 2889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:33.023142 2889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.028638 2889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40581
I20260620 14:31:33.028681 3017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40581 every 8 connection(s)
I20260620 14:31:33.028985 2889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:33.029765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 2889
I20260620 14:31:33.029865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:33.030047 392 external_mini_cluster.cc:1428] Setting key 4e0f19faa11103548a83869adb9552b9
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:31:33.039114 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.029759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35141 (local address 127.0.98.62:46463)
0620 14:31:33.029826 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:33.029828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:33.030043 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:31:33.031026 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:33.031028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:33.031042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:33.031094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:33.031712 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.032217 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.033251 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.033559 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.033654 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.036062 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:33.036063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:33.036064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:33.036071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:33.037650 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.038083 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.038084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.038084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.038143 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.038415 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.038416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.038416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.038529 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:33.038601 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:33.038920 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:31:33.039021 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:33.039268 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.029739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46463 (local address 127.0.98.1:35141)
0620 14:31:33.029900 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:33.029946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:33.030913 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:33.031176 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:33.031180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:33.031425 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:33.031616 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.031621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.032318 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.032319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.033146 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.033148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.033664 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.033665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.033835 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.034350 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:33.034358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:33.035960 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:33.037743 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.037750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.037752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.037993 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.038229 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.038230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.038231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.038339 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.038622 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:33.038628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:33.038784 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:31:33.038899 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:31:33.039048 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260620 14:31:33.039774 3018 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46463
I20260620 14:31:33.039858 3018 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:33.040069 3018 heartbeater.cc:511] Master 127.0.98.62:46463 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:31:33.040654 2827 ts_manager.cc:194] Registered new tserver with Master: f5539d9aaea94c879db4fdca757261c7 (127.0.98.1:40581)
I20260620 14:31:33.041589 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:33.054376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:33.133632 3023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:33.133857 3023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:33.133893 3023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:33.135040 3023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:33.135106 3023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:33.135150 3023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:33.136744 3023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:33.136979 3023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:33.137200 3023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:33.142084 3029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:33.142086 3031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:33.142082 3028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:33.142129 3023 server_base.cc:1034] running on GCE node
I20260620 14:31:33.142561 3023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:33.142818 3023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:33.143975 3023 hybrid_clock.cc:648] HybridClock initialized: now 1781965893143956 us; error 33 us; skew 500 ppm
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:33.146476 3023 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:33.146939 3023 webserver.cc:460] Webserver started at http://127.0.98.2:40809/ using document root <none> and password file <none>
I20260620 14:31:33.147153 3023 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:33.147217 3023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:33.147363 3023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:33.148447 3023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "64c389e301fe4a098cd2282df5e24d7a"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "dff73e50ce22335edf5ccee1a7aa406b"
server_key_iv: "829a3ec81c0697e98aa784f7a0a32854"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.148811 3023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64c389e301fe4a098cd2282df5e24d7a"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "dff73e50ce22335edf5ccee1a7aa406b"
server_key_iv: "829a3ec81c0697e98aa784f7a0a32854"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.150154 3023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:33.150851 3038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.151015 3023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:33.151115 3023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "64c389e301fe4a098cd2282df5e24d7a"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "dff73e50ce22335edf5ccee1a7aa406b"
server_key_iv: "829a3ec81c0697e98aa784f7a0a32854"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.151187 3023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:33.165921 3023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:33.167766 3023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:33.167932 3023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:33.168165 3023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:33.168497 3023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:33.168560 3023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.168598 3023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:33.168617 3023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.174345 3023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39799
I20260620 14:31:33.174384 3151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39799 every 8 connection(s)
I20260620 14:31:33.174685 3023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:33.178838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3023
I20260620 14:31:33.178925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:33.179126 392 external_mini_cluster.cc:1428] Setting key f5dd147ae4081974f576e4cb8d806a41
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:33.184324 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.175364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38869 (local address 127.0.98.62:46463)
0620 14:31:33.175408 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:33.175410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:33.175605 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:31:33.176500 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:33.176501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:33.176510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:33.176568 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:33.177148 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.177741 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.178693 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.178999 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.179078 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.181397 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:33.181398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:33.181399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:33.181405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:33.182873 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.183381 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.183382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.183382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.183430 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.183711 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.183712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.183712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.183806 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:33.183856 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:33.184198 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:31:33.184261 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:31:33.184672 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.175320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46463 (local address 127.0.98.2:38869)
0620 14:31:33.175478 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:33.175517 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:33.176411 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:33.176639 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:33.176643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:33.176888 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:33.177078 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.177083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.177820 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.177821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.178610 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.178612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.179086 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.179087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.179207 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.179740 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:33.179749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:33.181292 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:33.182968 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.182975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.182977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.183300 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.183492 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.183493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.183494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.183639 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.183871 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:33.183876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:33.184058 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:31:33.184184 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:31:33.184363 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260620 14:31:33.185184 3152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46463
I20260620 14:31:33.185294 3152 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:33.185501 3152 heartbeater.cc:511] Master 127.0.98.62:46463 requested a full tablet report, sending...
I20260620 14:31:33.185845 2827 ts_manager.cc:194] Registered new tserver with Master: 64c389e301fe4a098cd2282df5e24d7a (127.0.98.2:39799)
I20260620 14:31:33.186205 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38869
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:33.203537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:33.283946 3157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:33.284137 3157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:33.284159 3157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:33.285203 3157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:33.285240 3157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:33.285295 3157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:33.286803 3157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:33.286999 3157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:33.287191 3157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:33.292054 3165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:33.292060 3163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:33.292088 3157 server_base.cc:1034] running on GCE node
W20260620 14:31:33.292101 3162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:33.292514 3157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:33.292721 3157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:33.293891 3157 hybrid_clock.cc:648] HybridClock initialized: now 1781965893293877 us; error 37 us; skew 500 ppm
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:33.296322 3157 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:33.296734 3157 webserver.cc:460] Webserver started at http://127.0.98.3:33155/ using document root <none> and password file <none>
I20260620 14:31:33.296958 3157 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:33.297034 3157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:33.297183 3157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:33.298252 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "b88d1114c7434de2b106dd25872aaf3e"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "f81ddad3a3017f1d8d8d4893396ad37f"
server_key_iv: "7ab0110c83bf34742ee2abe1a0f011bc"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.298615 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b88d1114c7434de2b106dd25872aaf3e"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "f81ddad3a3017f1d8d8d4893396ad37f"
server_key_iv: "7ab0110c83bf34742ee2abe1a0f011bc"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.299891 3157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:33.300598 3172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.300757 3157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:33.300851 3157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "b88d1114c7434de2b106dd25872aaf3e"
format_stamp: "Formatted at 2026-06-20 14:31:33 on dist-test-slave-tc2g"
server_key: "f81ddad3a3017f1d8d8d4893396ad37f"
server_key_iv: "7ab0110c83bf34742ee2abe1a0f011bc"
server_key_version: "encryptionkey@0"
I20260620 14:31:33.300935 3157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:33.322772 3157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:33.324640 3157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:33.324807 3157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:33.325049 3157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:33.325369 3157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:33.325431 3157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.325492 3157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:33.325511 3157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:33.331295 3157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46257
I20260620 14:31:33.331342 3285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46257 every 8 connection(s)
I20260620 14:31:33.331650 3157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:33.338260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3157
I20260620 14:31:33.338368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:33.338558 392 external_mini_cluster.cc:1428] Setting key d237f0f9892b5537a7a762b91340f955
I20260620 14:31:33.341336 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.332348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54051 (local address 127.0.98.62:46463)
0620 14:31:33.332392 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:33.332394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:33.332663 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:31:33.333698 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:33.333699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:33.333711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:33.333762 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:33.334380 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.334884 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.335909 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.336105 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.336208 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.338537 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:33.338539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:33.338539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:33.338547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:33.339977 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.340389 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.340390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.340390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.340451 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:33.340702 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:33.340703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:33.340703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:33.340822 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:33.340894 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:33.341172 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:31:33.341247 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:33.341586 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:33.332343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46463 (local address 127.0.98.3:54051)
0620 14:31:33.332512 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:33.332570 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:33.333594 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:33.333838 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:33.333842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:33.334107 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:33.334288 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.334292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.334979 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.334981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.335803 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:33.335805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:33.336214 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:33.336215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:33.336347 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:33.336826 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:33.336853 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:33.338438 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:33.340078 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.340085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.340087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.340304 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.340515 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:33.340516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:33.340516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:33.340633 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:33.340899 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:33.340905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:33.341038 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:31:33.341139 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:31:33.341293 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 14:31:33.342006 3286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46463
I20260620 14:31:33.342095 3286 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:33.342231 3286 heartbeater.cc:511] Master 127.0.98.62:46463 requested a full tablet report, sending...
I20260620 14:31:33.342545 2827 ts_manager.cc:194] Registered new tserver with Master: b88d1114c7434de2b106dd25872aaf3e (127.0.98.3:46257)
I20260620 14:31:33.342919 2827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54051
I20260620 14:31:33.350147 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:31:33.361049 2876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:33.360026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.0.98.62:46463)
0620 14:31:33.360086 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:33.360089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:33.360215 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:31:33.360301 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:33.360301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:33.360311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:33.360386 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:33.360977 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47966: BlockingRecv error: recv got EOF from 127.0.0.1:47966 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:31:33.361966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2889
I20260620 14:31:33.366444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3023
I20260620 14:31:33.370674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3157
I20260620 14:31:33.374842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 2795
2026-06-20T14:31:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1819 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[3298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[3298](info): set up 2 sockets
Jun 20 14:31:33 dist-test-slave-tc2g krb5kdc[3298](info): commencing operation
krb5kdc: starting...
W20260620 14:31:34.409235 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:34.524694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:34.603645 3311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:34.603842 3311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:34.603863 3311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:34.605018 3311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:34.605059 3311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:34.605075 3311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:34.605088 3311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:34.605139 3311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:34.606578 3311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:34.606823 3311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:34.607051 3311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:34.611711 3319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:34.611737 3317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:34.611757 3316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:34.611950 3311 server_base.cc:1034] running on GCE node
I20260620 14:31:34.612224 3311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:34.612511 3311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:34.613646 3311 hybrid_clock.cc:648] HybridClock initialized: now 1781965894613632 us; error 43 us; skew 500 ppm
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:34.620704 3311 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:34.621179 3311 webserver.cc:460] Webserver started at http://127.0.98.62:41089/ using document root <none> and password file <none>
I20260620 14:31:34.621520 3311 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:34.621600 3311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:34.621737 3311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:34.622797 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "b9bedbffa76541bc87b120031f3126a9"
format_stamp: "Formatted at 2026-06-20 14:31:34 on dist-test-slave-tc2g"
server_key: "6de3d81e56fc251e83d76f367cac5822"
server_key_iv: "1a738d770cbbcfd5dd3ef770bbb9afd2"
server_key_version: "encryptionkey@0"
I20260620 14:31:34.623157 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9bedbffa76541bc87b120031f3126a9"
format_stamp: "Formatted at 2026-06-20 14:31:34 on dist-test-slave-tc2g"
server_key: "6de3d81e56fc251e83d76f367cac5822"
server_key_iv: "1a738d770cbbcfd5dd3ef770bbb9afd2"
server_key_version: "encryptionkey@0"
I20260620 14:31:34.624428 3311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:34.625100 3326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:34.625267 3311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:34.625380 3311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "b9bedbffa76541bc87b120031f3126a9"
format_stamp: "Formatted at 2026-06-20 14:31:34 on dist-test-slave-tc2g"
server_key: "6de3d81e56fc251e83d76f367cac5822"
server_key_iv: "1a738d770cbbcfd5dd3ef770bbb9afd2"
server_key_version: "encryptionkey@0"
I20260620 14:31:34.625491 3311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:34.632700 3311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:34.634476 3311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:34.634632 3311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:34.638615 3311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34079
I20260620 14:31:34.638652 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34079 every 8 connection(s)
I20260620 14:31:34.639120 3311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:34.639259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3311
I20260620 14:31:34.639358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:34.639614 392 external_mini_cluster.cc:1428] Setting key 47c9f2347cd60f34a9fd451c56867208
I20260620 14:31:34.640220 3379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:34.643218 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9: Bootstrap starting.
I20260620 14:31:34.643868 3379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:34.644168 3379 log.cc:824] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:34.644901 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9: No bootstrap required, opened a new log
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:34.646807 3379 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:31:34.646991 3379 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:34.647058 3379 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bedbffa76541bc87b120031f3126a9, State: Initialized, Role: FOLLOWER
I20260620 14:31:34.647225 3379 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:31:34.647312 3379 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:34.647356 3379 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:34.647404 3379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:34.647995 3379 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:31:34.648121 3379 leader_election.cc:302] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9bedbffa76541bc87b120031f3126a9; no voters:
I20260620 14:31:34.648296 3379 leader_election.cc:288] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:34.648350 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:34.648555 3379 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:34.648559 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [term 1 LEADER]: Becoming Leader. State: Replica: b9bedbffa76541bc87b120031f3126a9, State: Running, Role: LEADER
I20260620 14:31:34.648727 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:31:34.649287 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9bedbffa76541bc87b120031f3126a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } } }
I20260620 14:31:34.649386 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:34.649711 3393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:34.649386 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bedbffa76541bc87b120031f3126a9. Latest consensus state: current_term: 1 leader_uuid: "b9bedbffa76541bc87b120031f3126a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bedbffa76541bc87b120031f3126a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } } }
I20260620 14:31:34.650250 3393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:34.650707 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:34.640096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.0.98.62:34079)
0620 14:31:34.640302 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:34.640333 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:34.640367 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:31:34.640791 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:34.640797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:34.640840 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:34.640950 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:34.641737 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:34.642615 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:34.643685 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:34.644152 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:34.644362 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:34.646628 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:34.646632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:34.646632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:34.646644 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:34.648646 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:34.649068 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:34.649071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:34.649071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:34.649169 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:34.649532 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:34.649533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:34.649533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:34.649846 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:34.649965 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:34.650258 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:31:34.650511 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260620 14:31:34.650269 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:34.651930 3393 catalog_manager.cc:1347] Generated new cluster ID: 1de8f45d67e24f39a336683b3815231b
I20260620 14:31:34.652015 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:31:34.652032 3393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:34.666057 3393 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:34.666641 3393 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:34.668081 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:34.663966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.0.98.62:34079)
0620 14:31:34.664003 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:34.664005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:34.664130 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:31:34.664194 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:34.664195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:34.664205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:34.664271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:34.664693 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:34.665453 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:34.666477 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:34.666770 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:34.666840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:34.667598 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:34.667599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:34.667599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:34.667817 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:34.667833 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:34.667839 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:34.667889 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:34.667967 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:31:34.668004 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:34.671053 3393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bedbffa76541bc87b120031f3126a9: Generated new TSK 0
I20260620 14:31:34.671196 3393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:34.678287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3311
2026-06-20T14:31:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1301 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3412](info): set up 2 sockets
Jun 20 14:31:34 dist-test-slave-tc2g krb5kdc[3412](info): commencing operation
krb5kdc: starting...
W20260620 14:31:35.709295 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:35 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:35.824155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46013
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:35.906257 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:35.906510 3425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:35.906548 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:35.907742 3425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:35.907812 3425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:35.907836 3425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:35.907859 3425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:35.907887 3425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:35.909377 3425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:35.909688 3425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:35.909914 3425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:35.914728 3431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:35.914736 3430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:35.914739 3433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:35.914767 3425 server_base.cc:1034] running on GCE node
I20260620 14:31:35.915313 3425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:35.915598 3425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:35.916761 3425 hybrid_clock.cc:648] HybridClock initialized: now 1781965895916752 us; error 38 us; skew 500 ppm
Jun 20 14:31:35 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:35.919195 3425 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:35.919678 3425 webserver.cc:460] Webserver started at http://127.0.98.62:45709/ using document root <none> and password file <none>
I20260620 14:31:35.919871 3425 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:35.919920 3425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:35.920011 3425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:35.921015 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "8ee3612df70e485093981baff3485091"
format_stamp: "Formatted at 2026-06-20 14:31:35 on dist-test-slave-tc2g"
server_key: "010bc628f661dddcf40c96b845a3c9f3"
server_key_iv: "78661543b37b4224d434b8a620161d7c"
server_key_version: "encryptionkey@0"
I20260620 14:31:35.921307 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ee3612df70e485093981baff3485091"
format_stamp: "Formatted at 2026-06-20 14:31:35 on dist-test-slave-tc2g"
server_key: "010bc628f661dddcf40c96b845a3c9f3"
server_key_iv: "78661543b37b4224d434b8a620161d7c"
server_key_version: "encryptionkey@0"
I20260620 14:31:35.922649 3425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:35.923324 3440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:35.923565 3425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:35.923681 3425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "8ee3612df70e485093981baff3485091"
format_stamp: "Formatted at 2026-06-20 14:31:35 on dist-test-slave-tc2g"
server_key: "010bc628f661dddcf40c96b845a3c9f3"
server_key_iv: "78661543b37b4224d434b8a620161d7c"
server_key_version: "encryptionkey@0"
I20260620 14:31:35.923767 3425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:35.964295 3425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:35.966774 3425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:35.966943 3425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:35.970965 3425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46013
I20260620 14:31:35.971004 3492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46013 every 8 connection(s)
I20260620 14:31:35.971380 3425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:35.972357 3493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:35.974575 3493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091: Bootstrap starting.
I20260620 14:31:35.975111 3493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:35.975340 3493 log.cc:824] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:35.976011 3493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091: No bootstrap required, opened a new log
I20260620 14:31:35.977643 3493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } }
I20260620 14:31:35.977780 3493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:35.977805 3493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3612df70e485093981baff3485091, State: Initialized, Role: FOLLOWER
I20260620 14:31:35.977952 3493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } }
I20260620 14:31:35.978037 3493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:35.978060 3493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:35.978088 3493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:35.978845 3493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } }
I20260620 14:31:35.978928 3493 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee3612df70e485093981baff3485091; no voters:
I20260620 14:31:35.979051 3493 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:35.979136 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:35.979302 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3612df70e485093981baff3485091, State: Running, Role: LEADER
I20260620 14:31:35.979328 3493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:35.979390 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } }
I20260620 14:31:35.979872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3425
I20260620 14:31:35.979969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:35.980156 392 external_mini_cluster.cc:1428] Setting key 2b21ec02dc4bf7f6de26bc926f89e3d9
I20260620 14:31:35.980530 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee3612df70e485093981baff3485091. Latest consensus state: current_term: 1 leader_uuid: "8ee3612df70e485093981baff3485091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } } }
I20260620 14:31:35.980636 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:35.980754 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee3612df70e485093981baff3485091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee3612df70e485093981baff3485091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46013 } } }
I20260620 14:31:35.980815 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:35.980839 3513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:35.981714 3513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:35.984277 3513 catalog_manager.cc:1347] Generated new cluster ID: 029137cd9fdc4a2d8c7691c8d2b805c7
I20260620 14:31:35.984329 3513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:35 dist-test-slave-tc2g krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:35.989071 3513 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:35.989816 3513 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:35.992050 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:35.980937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.98.62:46013)
0620 14:31:35.981154 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:35.981164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:35.981199 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:31:35.981316 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:35.981321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:35.981364 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:35.981537 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:35.982309 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:35.983813 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:35.984826 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:35.985496 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:35.985609 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:35.988388 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:35.988392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:35.988392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:35.988403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:35.990479 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:35.990874 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:35.990877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:35.990877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:35.990969 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:35.991190 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:35.991191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:35.991191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:35.991436 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:35.991566 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:35.991696 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:31:35.991829 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260620 14:31:36.003561 3513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee3612df70e485093981baff3485091: Generated new TSK 0
I20260620 14:31:36.003748 3513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:36.031880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:36.113749 3519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:36.113978 3519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:36.114014 3519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:36.115172 3519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:36.115239 3519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:36.115283 3519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:36.116707 3519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:36.116997 3519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:36.117275 3519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:36.122117 3527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:36.122155 3524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:36.122191 3519 server_base.cc:1034] running on GCE node
W20260620 14:31:36.122149 3525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:36.122494 3519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:36.122794 3519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:36.123970 3519 hybrid_clock.cc:648] HybridClock initialized: now 1781965896123958 us; error 28 us; skew 500 ppm
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:36.126600 3519 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:36.127077 3519 webserver.cc:460] Webserver started at http://127.0.98.1:38821/ using document root <none> and password file <none>
I20260620 14:31:36.127296 3519 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:36.127360 3519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:36.127523 3519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:36.128571 3519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "4dd0e47004d9481e8513c47254ea7554"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "8e68c3afe19e369d292b501c2d522aea"
server_key_iv: "ca25f5914a61e990cc0b0770c301f9cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.128932 3519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4dd0e47004d9481e8513c47254ea7554"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "8e68c3afe19e369d292b501c2d522aea"
server_key_iv: "ca25f5914a61e990cc0b0770c301f9cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.130347 3519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:36.131098 3534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.131263 3519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:36.131357 3519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "4dd0e47004d9481e8513c47254ea7554"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "8e68c3afe19e369d292b501c2d522aea"
server_key_iv: "ca25f5914a61e990cc0b0770c301f9cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.131436 3519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:36.149873 3519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:36.152266 3519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:36.152410 3519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:36.152644 3519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:36.152941 3519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:36.153020 3519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.153062 3519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:36.153088 3519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.159017 3519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37487
I20260620 14:31:36.159056 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37487 every 8 connection(s)
I20260620 14:31:36.159355 3519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:36.167045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3519
I20260620 14:31:36.167155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:36.167348 392 external_mini_cluster.cc:1428] Setting key a442e985cbb41cb703017a36077800c0
I20260620 14:31:36.168766 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.160004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54083 (local address 127.0.98.62:46013)
0620 14:31:36.160076 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:36.160079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:36.160263 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:31:36.160778 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:36.160779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:36.160791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:36.160847 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:36.161849 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.162360 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.163352 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.163574 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.163661 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.165722 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:36.165723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:36.165724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:36.165731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:36.167173 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.167675 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.167676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.167676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.167746 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.168038 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.168039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.168039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.168165 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:36.168245 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:36.168574 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:31:36.168678 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:36.168969 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.159966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46013 (local address 127.0.98.1:54083)
0620 14:31:36.160106 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:36.160164 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:36.160677 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:36.160924 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:36.160927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:36.161549 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:36.161739 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.161744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.162468 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.162468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.163261 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.163263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.163671 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.163672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.163789 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.164295 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:36.164303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:36.165629 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:36.167271 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.167277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.167279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.167599 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.167837 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.167837 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.167838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.167967 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.168238 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:36.168244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:36.168379 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:31:36.168548 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 14:31:36.168704 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260620 14:31:36.169500 3648 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46013
I20260620 14:31:36.169593 3648 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:36.169788 3648 heartbeater.cc:511] Master 127.0.98.62:46013 requested a full tablet report, sending...
I20260620 14:31:36.170344 3457 ts_manager.cc:194] Registered new tserver with Master: 4dd0e47004d9481e8513c47254ea7554 (127.0.98.1:37487)
I20260620 14:31:36.170882 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54083
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:36.191885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:36.274047 3653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:36.274255 3653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:36.274278 3653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:36.275372 3653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:36.275406 3653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:36.275432 3653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:36.276886 3653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:36.277076 3653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:36.277243 3653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:36.282092 3661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:36.282114 3658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:36.282104 3659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:36.282331 3653 server_base.cc:1034] running on GCE node
I20260620 14:31:36.282533 3653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:36.282764 3653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:36.283923 3653 hybrid_clock.cc:648] HybridClock initialized: now 1781965896283915 us; error 40 us; skew 500 ppm
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:36.286448 3653 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:36.286895 3653 webserver.cc:460] Webserver started at http://127.0.98.2:45527/ using document root <none> and password file <none>
I20260620 14:31:36.287120 3653 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:36.287185 3653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:36.287326 3653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:36.288352 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "83fb5ca4ebeb42ddadb50da7d36e87e3"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "914dde3da7a23d2042ab295d55fbb61b"
server_key_iv: "d69b712522aef1326de7b45746735cd5"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.288739 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83fb5ca4ebeb42ddadb50da7d36e87e3"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "914dde3da7a23d2042ab295d55fbb61b"
server_key_iv: "d69b712522aef1326de7b45746735cd5"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.290087 3653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:31:36.290809 3668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.290985 3653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:36.291090 3653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "83fb5ca4ebeb42ddadb50da7d36e87e3"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "914dde3da7a23d2042ab295d55fbb61b"
server_key_iv: "d69b712522aef1326de7b45746735cd5"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.291162 3653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:36.313122 3653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:36.315624 3653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:36.315789 3653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:36.316028 3653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:36.316340 3653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:36.316392 3653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.316435 3653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:36.316466 3653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.322486 3653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43891
I20260620 14:31:36.322530 3781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43891 every 8 connection(s)
I20260620 14:31:36.322849 3653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:36.326926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3653
I20260620 14:31:36.327035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:36.327220 392 external_mini_cluster.cc:1428] Setting key bb67f4178d88170a688103777fd19c31
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:36.333037 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.323560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49077 (local address 127.0.98.62:46013)
0620 14:31:36.323614 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:36.323616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:36.323812 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:31:36.324348 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:36.324350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:36.324362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:36.324423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:36.325378 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.326027 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.327339 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.327576 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.327665 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.330125 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:36.330126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:36.330126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:36.330134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:36.331585 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.332066 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.332067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.332068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.332134 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.332390 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.332391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.332391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.332502 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:36.332579 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:36.332865 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:31:36.332949 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:31:36.333238 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.323502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46013 (local address 127.0.98.2:49077)
0620 14:31:36.323660 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:36.323721 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:36.324241 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:36.324492 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:36.324496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:36.325068 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:36.325271 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.325275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.326132 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.326134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.327223 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.327226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.327681 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.327682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.327793 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.328346 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:36.328356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:36.330020 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:36.331711 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.331717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.331720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.331987 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.332202 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.332202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.332203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.332313 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.332570 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:36.332576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:36.332728 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:31:36.332832 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:31:36.332986 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260620 14:31:36.333775 3782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46013
I20260620 14:31:36.333879 3782 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:36.334115 3782 heartbeater.cc:511] Master 127.0.98.62:46013 requested a full tablet report, sending...
I20260620 14:31:36.334520 3457 ts_manager.cc:194] Registered new tserver with Master: 83fb5ca4ebeb42ddadb50da7d36e87e3 (127.0.98.2:43891)
I20260620 14:31:36.335086 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49077
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:36.352438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:36.432789 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:36.433007 3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:36.433035 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:36.434294 3787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:36.434345 3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:36.434371 3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:36.435845 3787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:36.436096 3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:36.436327 3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:36.441244 3787 server_base.cc:1034] running on GCE node
W20260620 14:31:36.441211 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:36.441216 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:36.441205 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:36.441627 3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:36.441912 3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:36.443115 3787 hybrid_clock.cc:648] HybridClock initialized: now 1781965896443083 us; error 44 us; skew 500 ppm
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:36.445734 3787 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:36.446250 3787 webserver.cc:460] Webserver started at http://127.0.98.3:36909/ using document root <none> and password file <none>
I20260620 14:31:36.446494 3787 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:36.446573 3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:36.446723 3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:36.447801 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "261461994363477286cfc8da462c20ca"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "9301edba0f5fa78d349d3f62bec92b60"
server_key_iv: "0826576363642022aaecce9225c18259"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.448177 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "261461994363477286cfc8da462c20ca"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "9301edba0f5fa78d349d3f62bec92b60"
server_key_iv: "0826576363642022aaecce9225c18259"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.449570 3787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:31:36.450371 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.450551 3787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:36.450659 3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "261461994363477286cfc8da462c20ca"
format_stamp: "Formatted at 2026-06-20 14:31:36 on dist-test-slave-tc2g"
server_key: "9301edba0f5fa78d349d3f62bec92b60"
server_key_iv: "0826576363642022aaecce9225c18259"
server_key_version: "encryptionkey@0"
I20260620 14:31:36.450742 3787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:36.471009 3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:36.472929 3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:36.473104 3787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:36.473341 3787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:36.473688 3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:36.473752 3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.473794 3787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:36.473821 3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:36.479986 3787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34035
I20260620 14:31:36.480060 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34035 every 8 connection(s)
I20260620 14:31:36.480386 3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:36.487459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3787
I20260620 14:31:36.487577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:36.487764 392 external_mini_cluster.cc:1428] Setting key b92bc79025758da71eb7154894e3014a
I20260620 14:31:36.490037 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.481157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57737 (local address 127.0.98.62:46013)
0620 14:31:36.481227 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:36.481230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:36.481445 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:31:36.481968 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:36.481969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:36.481979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:36.482028 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:36.482929 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.483427 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.484394 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.484569 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.484620 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.486876 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:36.486877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:36.486877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:36.486884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:36.488615 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.489019 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.489020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.489020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.489071 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.489329 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.489330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.489330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.489553 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:36.489624 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:36.489912 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:31:36.489975 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:36.490226 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.481143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46013 (local address 127.0.98.3:57737)
0620 14:31:36.481303 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:36.481339 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:36.481882 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:36.482090 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:36.482093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:36.482665 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:36.482853 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.482857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.483512 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.483513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.484320 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:36.484322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.484640 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.484641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:36.484738 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.485292 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:36.485300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:36.486786 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:36.488695 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.488701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.488703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.488936 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.489130 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:36.489130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:36.489130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:36.489231 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:36.489660 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:36.489666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:36.489784 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:31:36.489884 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:31:36.490003 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260620 14:31:36.490770 3916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46013
I20260620 14:31:36.490850 3916 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:36.490980 3916 heartbeater.cc:511] Master 127.0.98.62:46013 requested a full tablet report, sending...
I20260620 14:31:36.491300 3457 ts_manager.cc:194] Registered new tserver with Master: 261461994363477286cfc8da462c20ca (127.0.98.3:34035)
I20260620 14:31:36.491667 3457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57737
I20260620 14:31:36.499531 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965896, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:36.521734 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:36.514591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.98.62:46013)
0620 14:31:36.514655 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:36.514657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:36.514799 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:31:36.514865 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:36.514866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:36.514875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:36.514917 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:36.515429 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.515905 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.516597 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:36.516769 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:36.516817 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:36.519241 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:36.519243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:36.519243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:36.519249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:36.520671 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.520973 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.520974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.520974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.521022 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:36.521180 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:36.521180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:36.521180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:36.521319 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:36.521397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:36.521592 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:31:36.521668 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:31:36.521998 3457 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57610
I20260620 14:31:36.522444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3519
I20260620 14:31:36.527030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3653
I20260620 14:31:36.531454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3787
I20260620 14:31:36.535694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3425
2026-06-20T14:31:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1859 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3928](info): set up 2 sockets
Jun 20 14:31:36 dist-test-slave-tc2g krb5kdc[3928](info): commencing operation
krb5kdc: starting...
W20260620 14:31:37.573309 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:37 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:37.688241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:37.768457 3941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:37.768692 3941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:37.768714 3941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:37.769939 3941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:37.769981 3941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:37.769996 3941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:37.770009 3941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:37.770063 3941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:37.771489 3941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:37.771754 3941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:37.771976 3941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:37.776726 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:37.776741 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:37.776759 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:37.777077 3941 server_base.cc:1034] running on GCE node
I20260620 14:31:37.777341 3941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:37.777654 3941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:37.778806 3941 hybrid_clock.cc:648] HybridClock initialized: now 1781965897778777 us; error 47 us; skew 500 ppm
Jun 20 14:31:37 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:37.781185 3941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:37.781678 3941 webserver.cc:460] Webserver started at http://127.0.98.62:44799/ using document root <none> and password file <none>
I20260620 14:31:37.781903 3941 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:37.781947 3941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:37.782086 3941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:37.783166 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "82411f1658604582b4b9f65a39e56114"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "ba4b9ed935cdc551d02ba03c772bc278"
server_key_iv: "d0b87c28d1d819cbcc0112f0cbf6502d"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.783511 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "82411f1658604582b4b9f65a39e56114"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "ba4b9ed935cdc551d02ba03c772bc278"
server_key_iv: "d0b87c28d1d819cbcc0112f0cbf6502d"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.784817 3941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:37.785641 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:37.785835 3941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:37.785934 3941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "82411f1658604582b4b9f65a39e56114"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "ba4b9ed935cdc551d02ba03c772bc278"
server_key_iv: "d0b87c28d1d819cbcc0112f0cbf6502d"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.786003 3941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:37.796729 3941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:37.798444 3941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:37.798600 3941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:37.802595 3941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42239
I20260620 14:31:37.802639 4008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42239 every 8 connection(s)
I20260620 14:31:37.803009 3941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:37.803817 4009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:37.806064 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114: Bootstrap starting.
I20260620 14:31:37.806576 4009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:37.806815 4009 log.cc:824] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:37.807417 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114: No bootstrap required, opened a new log
I20260620 14:31:37.808959 4009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260620 14:31:37.809094 4009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:37.809116 4009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82411f1658604582b4b9f65a39e56114, State: Initialized, Role: FOLLOWER
I20260620 14:31:37.809262 4009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260620 14:31:37.809345 4009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:37.809369 4009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:37.809397 4009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:37.809921 4009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260620 14:31:37.810000 4009 leader_election.cc:302] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82411f1658604582b4b9f65a39e56114; no voters:
I20260620 14:31:37.810118 4009 leader_election.cc:288] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:37.810199 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:37.810387 4009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:37.810590 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [term 1 LEADER]: Becoming Leader. State: Replica: 82411f1658604582b4b9f65a39e56114, State: Running, Role: LEADER
I20260620 14:31:37.810696 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260620 14:31:37.811064 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82411f1658604582b4b9f65a39e56114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } } }
I20260620 14:31:37.811090 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82411f1658604582b4b9f65a39e56114. Latest consensus state: current_term: 1 leader_uuid: "82411f1658604582b4b9f65a39e56114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82411f1658604582b4b9f65a39e56114" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } } }
I20260620 14:31:37.811154 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:37.811172 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:37.811319 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:37.811820 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:37.813061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 3941
I20260620 14:31:37.813177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:37.813357 392 external_mini_cluster.cc:1428] Setting key 9061b4f31fe7ef7bfa018a165d01e852
I20260620 14:31:37.813495 4021 catalog_manager.cc:1347] Generated new cluster ID: 2d5321f615164ac781c9879f72573435
I20260620 14:31:37.813539 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:37 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:37.824901 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:37.813828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.98.62:42239)
0620 14:31:37.814181 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:37.814190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:37.814220 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:31:37.814333 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:37.814339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:37.814397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:37.814514 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:37.815321 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.816659 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.817772 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.818367 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.818463 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:37.821224 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:37.821228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:37.821228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:37.821242 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:37.823306 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:37.823660 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:37.823663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:37.823664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:37.823776 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:37.823969 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:37.823970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:37.823970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:37.824294 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:37.824451 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:37.824519 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:31:37.824666 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":67,"threads_started":1}
I20260620 14:31:37.827507 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:37.827975 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:37.839321 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82411f1658604582b4b9f65a39e56114: Generated new TSK 0
I20260620 14:31:37.839529 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:37.866628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:37.947865 4035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:37.948127 4035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:37.948163 4035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:37.949342 4035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:37.949411 4035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:37.949486 4035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:37.950989 4035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:37.951248 4035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:37.951478 4035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:37.956382 4040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:37.956375 4041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:37.956441 4043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:37.956383 4035 server_base.cc:1034] running on GCE node
I20260620 14:31:37.956948 4035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:37.957192 4035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:37.958351 4035 hybrid_clock.cc:648] HybridClock initialized: now 1781965897958344 us; error 35 us; skew 500 ppm
Jun 20 14:31:37 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:37.960952 4035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:37.961503 4035 webserver.cc:460] Webserver started at http://127.0.98.1:41633/ using document root <none> and password file <none>
I20260620 14:31:37.961742 4035 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:37.961809 4035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:37.961957 4035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:37.962983 4035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e8de76d2c054ff7a6ebe16677886b37"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "2f81392640829e8bcd3905e2a6665d2d"
server_key_iv: "a4a5ac5bfd90dd2edf62088458efe3f8"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.963336 4035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e8de76d2c054ff7a6ebe16677886b37"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "2f81392640829e8bcd3905e2a6665d2d"
server_key_iv: "a4a5ac5bfd90dd2edf62088458efe3f8"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.964690 4035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:37.965381 4050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:37.965556 4035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:37.965674 4035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "8e8de76d2c054ff7a6ebe16677886b37"
format_stamp: "Formatted at 2026-06-20 14:31:37 on dist-test-slave-tc2g"
server_key: "2f81392640829e8bcd3905e2a6665d2d"
server_key_iv: "a4a5ac5bfd90dd2edf62088458efe3f8"
server_key_version: "encryptionkey@0"
I20260620 14:31:37.965750 4035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:37.975291 4035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:37.977595 4035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:37.977756 4035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:37.977981 4035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:37.978286 4035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:37.978338 4035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:37.978380 4035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:37.978406 4035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:37.984027 4035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36895
I20260620 14:31:37.984107 4163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36895 every 8 connection(s)
I20260620 14:31:37.984424 4035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:31:37 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:37.991847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4035
I20260620 14:31:37.991978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:37.992205 392 external_mini_cluster.cc:1428] Setting key 05ab130c6aa8b4a1e7132fc88c4c7707
I20260620 14:31:37.994340 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:37.985193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49235 (local address 127.0.98.62:42239)
0620 14:31:37.985255 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:37.985257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:37.985461 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:31:37.986414 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:37.986414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:37.986426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:37.986493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:37.987064 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.987555 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.988551 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.988750 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.988803 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:37.991005 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:37.991006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:37.991006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:37.991012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:37.992726 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:37.993214 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:37.993215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:37.993216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:37.993275 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:37.993641 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:37.993642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:37.993642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:37.993787 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:37.993858 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:37.994117 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:31:37.994211 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:31:37.994493 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:37.985148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42239 (local address 127.0.98.1:49235)
0620 14:31:37.985305 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:37.985348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:37.986320 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:37.986548 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:37.986551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:37.986806 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:37.986993 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:37.986997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.987653 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.987653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:37.988439 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:37.988441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:37.988820 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:37.988821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:37.988920 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:37.989487 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:37.989500 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:37.990922 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:37.992810 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:37.992817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:37.992819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:37.993135 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:37.993391 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:37.993392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:37.993392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:37.993568 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:37.993857 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:37.993861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:37.993976 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:31:37.994054 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:31:37.994216 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260620 14:31:37.995065 4164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42239
I20260620 14:31:37.995160 4164 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:37.995350 4164 heartbeater.cc:511] Master 127.0.98.62:42239 requested a full tablet report, sending...
I20260620 14:31:37.995922 3973 ts_manager.cc:194] Registered new tserver with Master: 8e8de76d2c054ff7a6ebe16677886b37 (127.0.98.1:36895)
I20260620 14:31:37.997012 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49235
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:38.016307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:38.100076 4169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:38.100301 4169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:38.100328 4169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:38.101517 4169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:38.101568 4169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:38.101595 4169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:38.103118 4169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:38.103344 4169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:38.103541 4169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:38.109023 4177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:38.109114 4169 server_base.cc:1034] running on GCE node
W20260620 14:31:38.109127 4175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:38.109023 4174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:38.109510 4169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:38.109798 4169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:38.110963 4169 hybrid_clock.cc:648] HybridClock initialized: now 1781965898110948 us; error 47 us; skew 500 ppm
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:38.113725 4169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:38.114198 4169 webserver.cc:460] Webserver started at http://127.0.98.2:42459/ using document root <none> and password file <none>
I20260620 14:31:38.114449 4169 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:38.114522 4169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:38.114686 4169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:38.115784 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "0738d132f5fd42a18fdd85c70f70ff35"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "138d4834520de701bc915f5b01d8b240"
server_key_iv: "fc0e8356e3c1fe2208ebb64e572e8b9f"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.116161 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0738d132f5fd42a18fdd85c70f70ff35"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "138d4834520de701bc915f5b01d8b240"
server_key_iv: "fc0e8356e3c1fe2208ebb64e572e8b9f"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.117622 4169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:38.118398 4184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.118566 4169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:38.118669 4169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "0738d132f5fd42a18fdd85c70f70ff35"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "138d4834520de701bc915f5b01d8b240"
server_key_iv: "fc0e8356e3c1fe2208ebb64e572e8b9f"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.118757 4169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:38.147589 4169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:38.150247 4169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:38.150429 4169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:38.150698 4169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:38.151031 4169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:38.151086 4169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.151129 4169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:38.151168 4169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.157696 4169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45199
I20260620 14:31:38.157754 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45199 every 8 connection(s)
I20260620 14:31:38.158126 4169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:38.161764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4169
I20260620 14:31:38.161895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:38.162088 392 external_mini_cluster.cc:1428] Setting key 39a7621e7827cd2b96bb75712bf2986a
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:38.169167 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.158889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33191 (local address 127.0.98.62:42239)
0620 14:31:38.158964 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:38.158966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:38.159186 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:31:38.160203 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:38.160204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:38.160217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:38.160275 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:38.160967 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.161617 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.163056 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.163272 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.163372 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.165845 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:38.165847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:38.165847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:38.165855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:38.167353 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.167988 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.167989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.167989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.168053 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.168379 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.168379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.168380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.168593 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:38.168649 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:38.169017 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:31:38.169097 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:31:38.169481 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.158886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42239 (local address 127.0.98.2:33191)
0620 14:31:38.159062 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:38.159101 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:38.160060 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:38.160359 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:38.160363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:38.160642 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:38.160852 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:38.160857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.161719 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.161720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:38.162944 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:38.162946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.163382 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.163384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:38.163517 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.164131 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:38.164144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:38.165748 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:38.167489 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:38.167497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:38.167501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:38.167880 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:38.168139 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:38.168140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:38.168140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:38.168306 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:38.168663 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:38.168669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:38.168860 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 14:31:38.168989 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:31:38.169185 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260620 14:31:38.169997 4298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42239
I20260620 14:31:38.170117 4298 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:38.170310 4298 heartbeater.cc:511] Master 127.0.98.62:42239 requested a full tablet report, sending...
I20260620 14:31:38.170794 3973 ts_manager.cc:194] Registered new tserver with Master: 0738d132f5fd42a18fdd85c70f70ff35 (127.0.98.2:45199)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:31:38.171263 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:38.188503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:38.271406 4303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:38.271620 4303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:38.271642 4303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:38.272819 4303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:38.272864 4303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:38.272891 4303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:38.274510 4303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:38.274775 4303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:38.275012 4303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:38.279964 4311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:38.280019 4308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:38.279978 4309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:38.280151 4303 server_base.cc:1034] running on GCE node
I20260620 14:31:38.280447 4303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:38.280699 4303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:38.281888 4303 hybrid_clock.cc:648] HybridClock initialized: now 1781965898281883 us; error 38 us; skew 500 ppm
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:38.284353 4303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:38.284861 4303 webserver.cc:460] Webserver started at http://127.0.98.3:43051/ using document root <none> and password file <none>
I20260620 14:31:38.285097 4303 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:38.285168 4303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:38.285331 4303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:38.286437 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "c79103ec356c4eb9a6455f505324e740"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "4d2e81ae898e0265268d7b93a6975a37"
server_key_iv: "d58af5b1bd6903285957c113470080e9"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.286809 4303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c79103ec356c4eb9a6455f505324e740"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "4d2e81ae898e0265268d7b93a6975a37"
server_key_iv: "d58af5b1bd6903285957c113470080e9"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.288138 4303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:38.288899 4318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.289080 4303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:38.289178 4303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "c79103ec356c4eb9a6455f505324e740"
format_stamp: "Formatted at 2026-06-20 14:31:38 on dist-test-slave-tc2g"
server_key: "4d2e81ae898e0265268d7b93a6975a37"
server_key_iv: "d58af5b1bd6903285957c113470080e9"
server_key_version: "encryptionkey@0"
I20260620 14:31:38.289268 4303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:38.305658 4303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:38.307530 4303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:38.307696 4303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:38.307940 4303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:38.308250 4303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:38.308305 4303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.308347 4303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:38.308375 4303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:38.315140 4303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37903
I20260620 14:31:38.315186 4431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37903 every 8 connection(s)
I20260620 14:31:38.315514 4303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:38.323814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4303
I20260620 14:31:38.323941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:38.324185 392 external_mini_cluster.cc:1428] Setting key 6704ab84a3a4284f0ca751b98cbd701d
I20260620 14:31:38.324883 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.316197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42289 (local address 127.0.98.62:42239)
0620 14:31:38.316273 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:38.316275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:38.316509 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:31:38.317556 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:38.317557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:38.317569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:38.317623 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:38.318211 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.318708 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.319645 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.319830 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.319879 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.322003 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:38.322005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:38.322005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:38.322011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:38.323493 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.323859 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.323860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.323861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.323924 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.324165 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.324166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.324166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.324320 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:38.324391 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:38.324692 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:31:38.324776 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:38.325119 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.316192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42239 (local address 127.0.98.3:42289)
0620 14:31:38.316385 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:38.316423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:38.317374 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:38.317682 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:38.317686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:38.317948 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:38.318140 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:38.318144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.318789 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.318789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:38.319576 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:38.319578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.319900 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.319901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:38.319996 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.320465 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:38.320473 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:38.321921 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:38.323561 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:38.323565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:38.323567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:38.323789 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:38.323989 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:38.323989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:38.323990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:38.324101 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:38.324382 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:38.324389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:38.324541 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:31:38.324652 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:31:38.324853 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260620 14:31:38.325574 4432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42239
I20260620 14:31:38.325657 4432 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:38.325863 4432 heartbeater.cc:511] Master 127.0.98.62:42239 requested a full tablet report, sending...
I20260620 14:31:38.326220 3973 ts_manager.cc:194] Registered new tserver with Master: c79103ec356c4eb9a6455f505324e740 (127.0.98.3:37903)
I20260620 14:31:38.326592 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42289
I20260620 14:31:38.336073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:38.358374 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.350870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.0.98.1:36895)
0620 14:31:38.351025 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:38.351031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:38.351050 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:31:38.351078 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:38.351080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:38.351094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:38.351185 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:38.351743 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.352451 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.353177 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.353564 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.353621 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.355923 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:38.355931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:38.355931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:38.355938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:38.357362 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.357725 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.357728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.357728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.357795 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.357943 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.357944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.357944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.358064 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:38.358173 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:38.358233 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:31:38.358293 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[3928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:38.365938 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:38.359692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.98.62:42239)
0620 14:31:38.359751 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:38.359753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:38.359762 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:38.359785 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:38.359786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:38.359803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:38.359853 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:38.360280 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.360752 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.361380 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:38.361636 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:38.361691 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:38.363657 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:38.363658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:38.363659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:38.363665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:38.365011 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.365294 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.365294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.365295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.365355 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:38.365544 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:38.365544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:38.365545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:38.365676 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:38.365746 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:38.365825 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:31:38.365877 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 14:31:38.366161 3973 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362
I20260620 14:31:38.366505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4035
I20260620 14:31:38.371057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4169
I20260620 14:31:38.375221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4303
I20260620 14:31:38.379528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 3941
2026-06-20T14:31:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1844 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[4444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[4444](info): set up 2 sockets
Jun 20 14:31:38 dist-test-slave-tc2g krb5kdc[4444](info): commencing operation
krb5kdc: starting...
W20260620 14:31:39.433555 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:39.550298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:39.631762 4457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:39.632018 4457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:39.632057 4457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:39.633270 4457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:39.633324 4457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:39.633370 4457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:39.633400 4457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:39.633468 4457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:39.634948 4457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:39.635136 4457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:39.635303 4457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:39.640021 4463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.640044 4465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.640028 4462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:39.640126 4457 server_base.cc:1034] running on GCE node
I20260620 14:31:39.640470 4457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:39.640714 4457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:39.641894 4457 hybrid_clock.cc:648] HybridClock initialized: now 1781965899641885 us; error 29 us; skew 500 ppm
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:39.653393 4457 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:39.653857 4457 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:39.653931 4457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:39.654083 4457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:39.655200 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "5d0ac2bc916d4937a9079e0b4fad59cb"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "34f3ef080f5d774e15442725ebf3929d"
server_key_iv: "ba330e972ffe675a56659c8c83b861df"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.655591 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d0ac2bc916d4937a9079e0b4fad59cb"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "34f3ef080f5d774e15442725ebf3929d"
server_key_iv: "ba330e972ffe675a56659c8c83b861df"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.656988 4457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:39.657786 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:39.657980 4457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:39.658102 4457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "5d0ac2bc916d4937a9079e0b4fad59cb"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "34f3ef080f5d774e15442725ebf3929d"
server_key_iv: "ba330e972ffe675a56659c8c83b861df"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.658177 4457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:39.680367 4457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:39.682796 4457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:39.682961 4457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:39.687198 4457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46799
I20260620 14:31:39.687225 4523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46799 every 8 connection(s)
I20260620 14:31:39.687520 4457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:39.688413 4524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:39.690855 4524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: Bootstrap starting.
I20260620 14:31:39.691469 4524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:39.691779 4524 log.cc:824] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:39.692478 4524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: No bootstrap required, opened a new log
I20260620 14:31:39.694090 4524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } }
I20260620 14:31:39.694224 4524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:39.694247 4524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0ac2bc916d4937a9079e0b4fad59cb, State: Initialized, Role: FOLLOWER
I20260620 14:31:39.694412 4524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } }
I20260620 14:31:39.694484 4524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:39.694505 4524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:39.694572 4524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:39.695216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4457
I20260620 14:31:39.695324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:39.695456 4524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } }
I20260620 14:31:39.695531 392 external_mini_cluster.cc:1428] Setting key 1ed9c52225775d643f6e0d0fc1d9b8b7
I20260620 14:31:39.695560 4524 leader_election.cc:302] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d0ac2bc916d4937a9079e0b4fad59cb; no voters:
I20260620 14:31:39.695773 4524 leader_election.cc:288] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:39.695807 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:39.695919 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [term 1 LEADER]: Becoming Leader. State: Replica: 5d0ac2bc916d4937a9079e0b4fad59cb, State: Running, Role: LEADER
I20260620 14:31:39.696043 4524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:39.696094 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } }
I20260620 14:31:39.696513 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } } }
I20260620 14:31:39.696604 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:39.696527 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d0ac2bc916d4937a9079e0b4fad59cb. Latest consensus state: current_term: 1 leader_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0ac2bc916d4937a9079e0b4fad59cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46799 } } }
I20260620 14:31:39.696646 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:39.697584 4538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:39.698228 4538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:39.699990 4538 catalog_manager.cc:1347] Generated new cluster ID: 02b436f679a64a87a8ee0b6228170626
I20260620 14:31:39.700058 4538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:39.707193 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:39.696139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.0.98.62:46799)
0620 14:31:39.696511 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:39.696520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:39.696554 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:31:39.696670 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:39.696678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:39.696739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:39.696879 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:39.697793 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.699136 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.699967 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.700591 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.700715 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:39.703470 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:39.703473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:39.703474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:39.703485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:39.705559 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:39.705930 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:39.705934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:39.705934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:39.706048 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:39.706254 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:39.706255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:39.706255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:39.706560 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:39.706750 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:39.706821 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:31:39.706958 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":74,"threads_started":1}
I20260620 14:31:39.707917 4538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:39.708410 4538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:39.715541 4538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: Generated new TSK 0
I20260620 14:31:39.715724 4538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:39.745486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:39.839666 4550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:39.839862 4550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:39.839917 4550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:39.841059 4550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:39.841099 4550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:39.841126 4550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:39.842648 4550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:39.842877 4550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:39.843128 4550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:39.848019 4556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.848019 4558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.848205 4555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:39.848181 4550 server_base.cc:1034] running on GCE node
I20260620 14:31:39.848448 4550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:39.848764 4550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:39.849928 4550 hybrid_clock.cc:648] HybridClock initialized: now 1781965899849907 us; error 37 us; skew 500 ppm
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:39.852383 4550 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:39.852792 4550 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:39.852864 4550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:39.853014 4550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:39.854133 4550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "15d92f871aa14789bb51ba638484a618"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "d40d58249756b9fd44153a7b7260f8f1"
server_key_iv: "ee2c57945b77cc08e68f26524b59f38e"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.854491 4550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "15d92f871aa14789bb51ba638484a618"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "d40d58249756b9fd44153a7b7260f8f1"
server_key_iv: "ee2c57945b77cc08e68f26524b59f38e"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.855875 4550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:39.856647 4564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:39.856838 4550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:39.856938 4550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "15d92f871aa14789bb51ba638484a618"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "d40d58249756b9fd44153a7b7260f8f1"
server_key_iv: "ee2c57945b77cc08e68f26524b59f38e"
server_key_version: "encryptionkey@0"
I20260620 14:31:39.857023 4550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:39.867718 4550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:39.869686 4550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:39.869856 4550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:39.870097 4550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:39.870419 4550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:39.870474 4550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:39.870509 4550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:39.870536 4550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:39.876441 4550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45011
I20260620 14:31:39.876483 4677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45011 every 8 connection(s)
I20260620 14:31:39.876745 4550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:39.880645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4550
I20260620 14:31:39.880776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:39.880952 392 external_mini_cluster.cc:1428] Setting key fe27720ebd7c93d76e3f1051584ad2db
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:39.886335 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:39.877444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45617 (local address 127.0.98.62:46799)
0620 14:31:39.877508 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:39.877511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:39.877730 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:31:39.878240 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:39.878241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:39.878257 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:39.878313 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:39.879218 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.879727 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.880730 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.881012 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.881096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:39.883538 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:39.883539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:39.883540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:39.883546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:39.884861 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:39.885350 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:39.885351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:39.885352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:39.885396 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:39.885721 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:39.885722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:39.885722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:39.885822 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:39.885873 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:39.886215 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:31:39.886275 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:31:39.886672 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:39.877410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46799 (local address 127.0.98.1:45617)
0620 14:31:39.877588 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:39.877623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:39.878133 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:39.878385 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:39.878388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:39.878635 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:39.879111 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:39.879116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.879831 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.879832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:39.880612 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:39.880614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:39.881109 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:39.881110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:39.881291 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:39.881736 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:39.881745 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:39.883429 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:39.884944 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:39.884952 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:39.884954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:39.885276 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:39.885497 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:39.885498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:39.885498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:39.885638 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:39.885890 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:39.885897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:39.886069 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 14:31:39.886198 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:31:39.886380 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260620 14:31:39.887218 4678 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46799
I20260620 14:31:39.887308 4678 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:39.887466 4678 heartbeater.cc:511] Master 127.0.98.62:46799 requested a full tablet report, sending...
I20260620 14:31:39.887945 4488 ts_manager.cc:194] Registered new tserver with Master: 15d92f871aa14789bb51ba638484a618 (127.0.98.1:45011)
I20260620 14:31:39.888541 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45617
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:39.905823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:39.985179 4683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:39.985423 4683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:39.985488 4683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:39.986678 4683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:39.986748 4683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:39.986792 4683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:39.988315 4683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:39.988610 4683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:39.988852 4683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:39.993731 4691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.993731 4689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:39.993775 4688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:39.993997 4683 server_base.cc:1034] running on GCE node
I20260620 14:31:39.994241 4683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:39.994449 4683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:39.995590 4683 hybrid_clock.cc:648] HybridClock initialized: now 1781965899995565 us; error 48 us; skew 500 ppm
Jun 20 14:31:39 dist-test-slave-tc2g krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:39.998381 4683 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:39.998752 4683 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:39.998801 4683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:39.998890 4683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:39.999925 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "2562eff37c8f46ee88a17578da3faf68"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "2aa5592209891274a151cd4f85ba9c90"
server_key_iv: "658d7f3de2a5760bf64a55ae7a3122db"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.000233 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2562eff37c8f46ee88a17578da3faf68"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "2aa5592209891274a151cd4f85ba9c90"
server_key_iv: "658d7f3de2a5760bf64a55ae7a3122db"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.001492 4683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:40.002153 4697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.002327 4683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:40.002394 4683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "2562eff37c8f46ee88a17578da3faf68"
format_stamp: "Formatted at 2026-06-20 14:31:39 on dist-test-slave-tc2g"
server_key: "2aa5592209891274a151cd4f85ba9c90"
server_key_iv: "658d7f3de2a5760bf64a55ae7a3122db"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.002444 4683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.011401 4683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.013842 4683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.013963 4683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.014158 4683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:40.014489 4683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:40.014525 4683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.014571 4683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:40.014598 4683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.020478 4683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36039
I20260620 14:31:40.020514 4810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36039 every 8 connection(s)
I20260620 14:31:40.020797 4683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:31:40 dist-test-slave-tc2g krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:40.029285 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.021499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53305 (local address 127.0.98.62:46799)
0620 14:31:40.021551 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.021553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.021768 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:31:40.022227 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.022228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.022238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.022287 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.023172 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.023654 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.024579 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.024764 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.024812 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.026953 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.026954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.026954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:40.026960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.028166 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.028510 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.028511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.028511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.028568 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.028788 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.028789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.028789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.028887 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.028935 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.029154 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.029223 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:40.029474 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.021484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46799 (local address 127.0.98.2:53305)
0620 14:31:40.021630 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.021684 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.022142 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.022350 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.022353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.022601 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.023099 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.023104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.023731 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.023732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.024510 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.024511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.024832 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.024833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.024928 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.025507 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:40.025516 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.026868 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.028236 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:40.028241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:40.028243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:40.028448 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:40.028625 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:40.028625 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:40.028626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:40.028735 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:40.028940 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.028944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.029053 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.029131 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.029247 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 14:31:40.030009 4811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46799
I20260620 14:31:40.030108 4811 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:40.030282 4811 heartbeater.cc:511] Master 127.0.98.62:46799 requested a full tablet report, sending...
I20260620 14:31:40.030342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4683
I20260620 14:31:40.030431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:40.030635 392 external_mini_cluster.cc:1428] Setting key 008f730823a3385e8b7be765af90b6ba
I20260620 14:31:40.030640 4488 ts_manager.cc:194] Registered new tserver with Master: 2562eff37c8f46ee88a17578da3faf68 (127.0.98.2:36039)
I20260620 14:31:40.031231 4488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53305
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:40.054476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:40.135550 4816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:40.135746 4816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:40.135767 4816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:40.137125 4816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:40.137184 4816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:40.137223 4816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:40.138837 4816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:40.139027 4816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:40.139194 4816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:40.143949 4824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.143957 4822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.143953 4821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.144187 4816 server_base.cc:1034] running on GCE node
I20260620 14:31:40.144403 4816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:40.144582 4816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:40.145728 4816 hybrid_clock.cc:648] HybridClock initialized: now 1781965900145711 us; error 40 us; skew 500 ppm
Jun 20 14:31:40 dist-test-slave-tc2g krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:40.148217 4816 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:40.148494 4816 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:40.148550 4816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:40.148715 4816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:40.149803 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc4715ba94b04551b7043f8ba841dd88"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "d5f291d26804f92633e5cc8d8c7d7079"
server_key_iv: "6e3af8aee494f51292e2a97c40be7e7f"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.150159 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc4715ba94b04551b7043f8ba841dd88"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "d5f291d26804f92633e5cc8d8c7d7079"
server_key_iv: "6e3af8aee494f51292e2a97c40be7e7f"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.151443 4816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:40.152122 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.152287 4816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:40.152397 4816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "cc4715ba94b04551b7043f8ba841dd88"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "d5f291d26804f92633e5cc8d8c7d7079"
server_key_iv: "6e3af8aee494f51292e2a97c40be7e7f"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.152468 4816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.166131 4816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.168404 4816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.168569 4816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.168802 4816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:40.169106 4816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:40.169158 4816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.169193 4816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:40.169220 4816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.174834 4816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36257
I20260620 14:31:40.174906 4943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36257 every 8 connection(s)
I20260620 14:31:40.175180 4816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:40.179072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4816
I20260620 14:31:40.179193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:40.179395 392 external_mini_cluster.cc:1428] Setting key ffd8bbf8422ed30c19cfe6a7a6575a53
Jun 20 14:31:40 dist-test-slave-tc2g krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:40.184312 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.175901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56149 (local address 127.0.98.62:46799)
0620 14:31:40.175963 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.175965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.176139 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:31:40.176591 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.176592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.176600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.176654 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.177609 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.178114 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.179147 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.179452 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.179522 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.181750 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.181751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.181752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:40.181757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.183159 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.183501 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.183501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.183502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.183549 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.183819 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.183820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.183820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.183912 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.184000 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.184194 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.184253 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:31:40.184511 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.175857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46799 (local address 127.0.98.3:56149)
0620 14:31:40.176012 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.176052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.176513 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.176733 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.176736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.176978 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.177523 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.177528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.178218 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.178219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.179050 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.179052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.179546 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.179547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.179691 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.180232 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:40.180240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.181673 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.183234 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:40.183240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:40.183241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:40.183438 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:40.183634 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:40.183635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:40.183636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:40.183758 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:40.183967 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.183970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.184088 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.184166 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.184279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 14:31:40.184930 4944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46799
I20260620 14:31:40.185005 4944 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:40.185108 4944 heartbeater.cc:511] Master 127.0.98.62:46799 requested a full tablet report, sending...
I20260620 14:31:40.185454 4487 ts_manager.cc:194] Registered new tserver with Master: cc4715ba94b04551b7043f8ba841dd88 (127.0.98.3:36257)
I20260620 14:31:40.185794 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56149
I20260620 14:31:40.191265 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:40.198724 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.192845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.0.98.62:46799)
0620 14:31:40.192883 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.192884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.192992 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:31:40.193041 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.193042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.193049 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.193099 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.193724 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.194169 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.194836 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.194997 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.195043 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.196119 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.196120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.196121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:40.196126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.197682 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.198021 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.198022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.198022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.198067 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:40.198226 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:40.198226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:40.198226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:40.198344 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.198429 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.198599 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.198655 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:40.200466 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:40.202129 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:40.208532 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.0.98.1:45011)
0620 14:31:40.204585 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.204592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.204606 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:40.204638 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.204639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.204672 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.204766 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:40.204934 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206094 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.207567 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208092 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208183 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208243 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.208411 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.208447 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":73,"threads_started":1}
I20260620 14:31:40.208521 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45011 (local address 127.0.0.1:38014)
0620 14:31:40.204332 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.204376 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.204446 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.204789 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.204792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.204795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:40.204863 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.204865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206191 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.206192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.207471 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.207473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208185 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.208275 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208290 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.208393 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.208447 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":38,"thread_start_us":58,"threads_started":1}
I20260620 14:31:40.208714 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.2:36039)
0620 14:31:40.204912 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.204918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.204957 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:31:40.204993 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.204995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.205009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.205103 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:40.205208 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206127 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.207496 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208183 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208322 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.208585 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.208633 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":89,"threads_started":1}
I20260620 14:31:40.208781 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36257 (local address 127.0.0.1:53662)
0620 14:31:40.204705 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.204713 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.204788 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.205315 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.205316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.205318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:40.205355 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.205357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206561 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.206562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.207606 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.207608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208517 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.208591 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208594 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.208641 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.208674 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":44,"threads_started":1}
I20260620 14:31:40.208860 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36039 (local address 127.0.0.1:51032)
0620 14:31:40.204437 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.204448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.204497 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.205099 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.205100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.205102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:40.205147 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.205148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206231 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.206232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.207410 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.207412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208358 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.208429 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208433 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.208480 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.208522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":63,"thread_start_us":77,"threads_started":1}
I20260620 14:31:40.209088 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.204558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.0.98.3:36257)
0620 14:31:40.204738 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.204743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.204777 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:31:40.204851 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.204853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.204868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.205228 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:40.205445 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.206464 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.207683 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.208379 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.208438 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.208487 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.208943 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.208991 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260620 14:31:40.209685 4745 tablet_service.cc:1467] Processing CreateTablet for tablet e32635d3d837464886cb073c9ca2efa4 (DEFAULT_TABLE table=test-table [id=731917edfec9438bb1d02976c1925d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.209728 4878 tablet_service.cc:1467] Processing CreateTablet for tablet e32635d3d837464886cb073c9ca2efa4 (DEFAULT_TABLE table=test-table [id=731917edfec9438bb1d02976c1925d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.209964 4745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32635d3d837464886cb073c9ca2efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.210067 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32635d3d837464886cb073c9ca2efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.212339 4959 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Bootstrap starting.
I20260620 14:31:40.212349 4958 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Bootstrap starting.
I20260620 14:31:40.212987 4958 tablet_bootstrap.cc:654] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.213004 4959 tablet_bootstrap.cc:654] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.213189 4612 tablet_service.cc:1467] Processing CreateTablet for tablet e32635d3d837464886cb073c9ca2efa4 (DEFAULT_TABLE table=test-table [id=731917edfec9438bb1d02976c1925d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.213587 4612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32635d3d837464886cb073c9ca2efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.213611 4958 log.cc:824] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.213604 4959 log.cc:824] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.214289 4959 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: No bootstrap required, opened a new log
I20260620 14:31:40.214370 4958 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: No bootstrap required, opened a new log
I20260620 14:31:40.214372 4959 ts_tablet_manager.cc:1397] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:40.214617 4958 ts_tablet_manager.cc:1397] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:31:40.215984 4959 raft_consensus.cc:348] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.216136 4959 raft_consensus.cc:374] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.216197 4959 raft_consensus.cc:729] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2562eff37c8f46ee88a17578da3faf68, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.216351 4959 consensus_queue.cc:260] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.216629 4959 ts_tablet_manager.cc:1428] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:40.216761 4811 heartbeater.cc:503] Master 127.0.98.62:46799 was elected leader, sending a full tablet report...
I20260620 14:31:40.216780 4958 raft_consensus.cc:348] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.217002 4958 raft_consensus.cc:374] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.217092 4958 raft_consensus.cc:729] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4715ba94b04551b7043f8ba841dd88, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.217116 4963 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Bootstrap starting.
I20260620 14:31:40.217217 4958 consensus_queue.cc:260] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.217492 4958 ts_tablet_manager.cc:1428] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:31:40.217583 4944 heartbeater.cc:503] Master 127.0.98.62:46799 was elected leader, sending a full tablet report...
I20260620 14:31:40.217782 4963 tablet_bootstrap.cc:654] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.218389 4963 log.cc:824] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.219141 4963 tablet_bootstrap.cc:492] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: No bootstrap required, opened a new log
I20260620 14:31:40.219216 4963 ts_tablet_manager.cc:1397] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:31:40.220559 4963 raft_consensus.cc:348] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.220708 4963 raft_consensus.cc:374] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.220753 4963 raft_consensus.cc:729] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d92f871aa14789bb51ba638484a618, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.220862 4963 consensus_queue.cc:260] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.221110 4963 ts_tablet_manager.cc:1428] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:31:40.221138 4678 heartbeater.cc:503] Master 127.0.98.62:46799 was elected leader, sending a full tablet report...
W20260620 14:31:40.271323 4812 tablet.cc:2369] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:40.361370 4966 raft_consensus.cc:482] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:40.361541 4966 raft_consensus.cc:504] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.361989 4966 leader_election.cc:288] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc4715ba94b04551b7043f8ba841dd88 (127.0.98.3:36257), 2562eff37c8f46ee88a17578da3faf68 (127.0.98.2:36039)
I20260620 14:31:40.364683 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.362184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36039 (local address 127.0.98.1:37743)
0620 14:31:40.362252 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.362263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.362317 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.362465 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.362465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.362467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:40.362543 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.362545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.363167 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.363168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.363895 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.363896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.364426 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.364428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.364505 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.364508 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.364573 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.364611 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260620 14:31:40.364681 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.362285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37743 (local address 127.0.98.2:36039)
0620 14:31:40.362337 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.362339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.362348 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:40.362376 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.362377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.362387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.362461 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:40.362616 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.363078 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.363989 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.364336 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.364426 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.364436 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.364578 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.364613 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:31:40.364807 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.362213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36257 (local address 127.0.98.1:37287)
0620 14:31:40.362368 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.362377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.362457 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.362597 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.362598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.362600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:40.362667 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.362676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.363268 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.363269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.363980 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:40.363981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.364534 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.364534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:40.364606 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.364609 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.364694 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.364730 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260620 14:31:40.364800 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.362263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37287 (local address 127.0.98.3:36257)
0620 14:31:40.362313 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.362315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.362447 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:31:40.362513 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.362514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.362524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.362595 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:40.362751 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.363186 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.364075 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.364441 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.364533 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.364542 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.364705 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.364741 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:31:40.365016 4765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32635d3d837464886cb073c9ca2efa4" candidate_uuid: "15d92f871aa14789bb51ba638484a618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2562eff37c8f46ee88a17578da3faf68" is_pre_election: true
I20260620 14:31:40.365118 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32635d3d837464886cb073c9ca2efa4" candidate_uuid: "15d92f871aa14789bb51ba638484a618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4715ba94b04551b7043f8ba841dd88" is_pre_election: true
I20260620 14:31:40.365191 4765 raft_consensus.cc:2454] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d92f871aa14789bb51ba638484a618 in term 0.
I20260620 14:31:40.365283 4898 raft_consensus.cc:2454] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d92f871aa14789bb51ba638484a618 in term 0.
I20260620 14:31:40.365443 4567 leader_election.cc:302] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d92f871aa14789bb51ba638484a618, 2562eff37c8f46ee88a17578da3faf68; no voters:
I20260620 14:31:40.365578 4966 raft_consensus.cc:2781] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:40.365633 4966 raft_consensus.cc:482] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:40.365665 4966 raft_consensus.cc:3037] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:40.366341 4966 raft_consensus.cc:504] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.366482 4966 leader_election.cc:288] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [CANDIDATE]: Term 1 election: Requested vote from peers cc4715ba94b04551b7043f8ba841dd88 (127.0.98.3:36257), 2562eff37c8f46ee88a17578da3faf68 (127.0.98.2:36039)
I20260620 14:31:40.366660 4765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32635d3d837464886cb073c9ca2efa4" candidate_uuid: "15d92f871aa14789bb51ba638484a618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2562eff37c8f46ee88a17578da3faf68"
I20260620 14:31:40.366729 4765 raft_consensus.cc:3037] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:40.367450 4765 raft_consensus.cc:2454] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d92f871aa14789bb51ba638484a618 in term 1.
I20260620 14:31:40.367563 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32635d3d837464886cb073c9ca2efa4" candidate_uuid: "15d92f871aa14789bb51ba638484a618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4715ba94b04551b7043f8ba841dd88"
I20260620 14:31:40.367619 4567 leader_election.cc:302] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d92f871aa14789bb51ba638484a618, 2562eff37c8f46ee88a17578da3faf68; no voters:
I20260620 14:31:40.367660 4898 raft_consensus.cc:3037] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:40.367766 4966 raft_consensus.cc:2781] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:40.367930 4966 raft_consensus.cc:686] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [term 1 LEADER]: Becoming Leader. State: Replica: 15d92f871aa14789bb51ba638484a618, State: Running, Role: LEADER
I20260620 14:31:40.368027 4966 consensus_queue.cc:237] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } }
I20260620 14:31:40.368573 4898 raft_consensus.cc:2454] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d92f871aa14789bb51ba638484a618 in term 1.
I20260620 14:31:40.368937 4488 catalog_manager.cc:5495] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15d92f871aa14789bb51ba638484a618 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "15d92f871aa14789bb51ba638484a618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d92f871aa14789bb51ba638484a618" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 } health_report { overall_health: UNKNOWN } } }
W20260620 14:31:40.377336 4679 tablet.cc:2369] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:40.391453 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.389022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.98.1:45011)
0620 14:31:40.389089 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.389091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.389101 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:40.389143 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.389144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.389165 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.389253 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:40.389398 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.389909 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.390662 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:40.390867 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:40.390942 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:40.391105 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:40.391216 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:40.391303 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.391332 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.391351 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:40.393206 4765 raft_consensus.cc:1261] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 1 FOLLOWER]: Refusing update from remote peer 15d92f871aa14789bb51ba638484a618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:40.393219 4898 raft_consensus.cc:1261] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 1 FOLLOWER]: Refusing update from remote peer 15d92f871aa14789bb51ba638484a618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:40.393559 4968 consensus_queue.cc:1035] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4715ba94b04551b7043f8ba841dd88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:40.393719 4966 consensus_queue.cc:1035] T e32635d3d837464886cb073c9ca2efa4 P 15d92f871aa14789bb51ba638484a618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2562eff37c8f46ee88a17578da3faf68" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:40.396051 4971 mvcc.cc:205] Tried to move back new op lower bound from 7298932328008003584 to 7298932327908171776. Current Snapshot: MvccSnapshot[applied={T|T < 7298932328008003584}]
I20260620 14:31:40.396781 4972 mvcc.cc:205] Tried to move back new op lower bound from 7298932328008003584 to 7298932327908171776. Current Snapshot: MvccSnapshot[applied={T|T < 7298932328008003584}]
I20260620 14:31:40.398769 4488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:40.398857 4488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:40.399593 4488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d0ac2bc916d4937a9079e0b4fad59cb: Sending DeleteTablet for 3 replicas of tablet e32635d3d837464886cb073c9ca2efa4
I20260620 14:31:40.399900 4745 tablet_service.cc:1514] Processing DeleteTablet for tablet e32635d3d837464886cb073c9ca2efa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51032
I20260620 14:31:40.399979 4878 tablet_service.cc:1514] Processing DeleteTablet for tablet e32635d3d837464886cb073c9ca2efa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53662
I20260620 14:31:40.400161 4612 tablet_service.cc:1514] Processing DeleteTablet for tablet e32635d3d837464886cb073c9ca2efa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38014
I20260620 14:31:40.400214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4550
I20260620 14:31:40.401227 4981 tablet_replica.cc:330] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: stopping tablet replica
I20260620 14:31:40.401345 4981 raft_consensus.cc:2229] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:40.401473 4981 raft_consensus.cc:2258] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:40.401871 4981 ts_tablet_manager.cc:1905] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:40.403090 4982 tablet_replica.cc:330] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: stopping tablet replica
I20260620 14:31:40.403218 4982 raft_consensus.cc:2229] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:40.403373 4982 raft_consensus.cc:2258] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:40.403376 4981 ts_tablet_manager.cc:1918] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:40.403438 4981 log.cc:1196] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/e32635d3d837464886cb073c9ca2efa4
I20260620 14:31:40.403816 4982 ts_tablet_manager.cc:1905] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:40.403846 4981 ts_tablet_manager.cc:1939] T e32635d3d837464886cb073c9ca2efa4 P 2562eff37c8f46ee88a17578da3faf68: Deleting consensus metadata
I20260620 14:31:40.404215 4474 catalog_manager.cc:4841] TS 2562eff37c8f46ee88a17578da3faf68 (127.0.98.2:36039): tablet e32635d3d837464886cb073c9ca2efa4 (table test-table [id=731917edfec9438bb1d02976c1925d17]) successfully deleted
W20260620 14:31:40.405627 4475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45011 (error 108)
I20260620 14:31:40.405664 4982 ts_tablet_manager.cc:1918] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:40.405741 4982 log.cc:1196] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/e32635d3d837464886cb073c9ca2efa4
I20260620 14:31:40.405836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4683
I20260620 14:31:40.406584 4982 ts_tablet_manager.cc:1939] T e32635d3d837464886cb073c9ca2efa4 P cc4715ba94b04551b7043f8ba841dd88: Deleting consensus metadata
I20260620 14:31:40.406996 4475 catalog_manager.cc:4841] TS cc4715ba94b04551b7043f8ba841dd88 (127.0.98.3:36257): tablet e32635d3d837464886cb073c9ca2efa4 (table test-table [id=731917edfec9438bb1d02976c1925d17]) successfully deleted
I20260620 14:31:40.407280 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.407152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45011 (local address 127.0.0.1:38040)
0620 14:31:40.407213 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.407255 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:31:40.407413 4475 catalog_manager.cc:4584] TS 15d92f871aa14789bb51ba638484a618 (127.0.98.1:45011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e32635d3d837464886cb073c9ca2efa4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45011: connect: Connection refused (error 111)
I20260620 14:31:40.410377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4816
I20260620 14:31:40.415714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4457
2026-06-20T14:31:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2035 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:40Z Disabled control of system clock
I20260620 14:31:40.442833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:31:40.521971 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:40.522186 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:40.522207 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:40.523325 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:40.523365 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:40.523378 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:40.523391 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:40.523442 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:40.524729 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:40.524969 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:40.525257 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:40.529870 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.529881 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.529886 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.530210 4988 server_base.cc:1034] running on GCE node
I20260620 14:31:40.530464 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:40.530788 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:40.531934 4988 hybrid_clock.cc:648] HybridClock initialized: now 1781965900531916 us; error 35 us; skew 500 ppm
I20260620 14:31:40.533191 4988 webserver.cc:460] Webserver started at http://127.0.98.62:33897/ using document root <none> and password file <none>
I20260620 14:31:40.533522 4988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:40.533598 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:40.533735 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:40.534862 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "010a6cab0c7c4a63a114dda46d3573db"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "42fca1ed802fc7bb86a8cb6fa9cc6f5a"
server_key_iv: "b55925c45611041003f377008a284bd8"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.535315 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "010a6cab0c7c4a63a114dda46d3573db"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "42fca1ed802fc7bb86a8cb6fa9cc6f5a"
server_key_iv: "b55925c45611041003f377008a284bd8"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.536645 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:40.537348 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.537515 4988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:40.537606 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "010a6cab0c7c4a63a114dda46d3573db"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "42fca1ed802fc7bb86a8cb6fa9cc6f5a"
server_key_iv: "b55925c45611041003f377008a284bd8"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.537686 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.540827 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.541086 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.541232 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.544922 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35223
I20260620 14:31:40.544950 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35223 every 8 connection(s)
I20260620 14:31:40.545289 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:40.546026 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.546685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 4988
I20260620 14:31:40.546782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:40.546942 392 external_mini_cluster.cc:1428] Setting key 68d68bc7aa05ed91ac82e14583e64570
I20260620 14:31:40.549157 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: Bootstrap starting.
I20260620 14:31:40.549579 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.547458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.98.62:35223)
0620 14:31:40.547682 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.547694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.547716 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:31:40.548122 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.548129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.548164 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.548316 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.549018 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.549021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.549022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.549026 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.549074 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.549084 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.549160 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.549226 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.549370 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 14:31:40.549912 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.550217 5055 log.cc:824] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.550854 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: No bootstrap required, opened a new log
I20260620 14:31:40.552160 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260620 14:31:40.552311 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.552359 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010a6cab0c7c4a63a114dda46d3573db, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.552484 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260620 14:31:40.552592 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:40.552635 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:40.552686 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:40.553581 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260620 14:31:40.553696 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 010a6cab0c7c4a63a114dda46d3573db; no voters:
I20260620 14:31:40.553859 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:40.553910 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:40.554054 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [term 1 LEADER]: Becoming Leader. State: Replica: 010a6cab0c7c4a63a114dda46d3573db, State: Running, Role: LEADER
I20260620 14:31:40.554152 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:40.554190 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260620 14:31:40.554606 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "010a6cab0c7c4a63a114dda46d3573db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260620 14:31:40.554716 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:40.554879 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 010a6cab0c7c4a63a114dda46d3573db. Latest consensus state: current_term: 1 leader_uuid: "010a6cab0c7c4a63a114dda46d3573db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6cab0c7c4a63a114dda46d3573db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260620 14:31:40.554978 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:40.555191 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:40.555653 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:40.557236 5068 catalog_manager.cc:1347] Generated new cluster ID: a656badf6edc4dfaa2d2a8235cbd4848
I20260620 14:31:40.557291 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:40.582935 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:40.583153 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:40.586840 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: Generated new TSK 0
I20260620 14:31:40.587038 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:40.593696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--builtin_ntp_servers=127.0.98.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:31:40.675081 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:40.675282 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:40.675303 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:40.676389 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:40.676424 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:40.676450 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:40.677731 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:40.677978 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:40.678211 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:40.682934 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.682925 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.682950 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.683152 5079 server_base.cc:1034] running on GCE node
I20260620 14:31:40.683377 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:40.683611 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:40.684772 5079 hybrid_clock.cc:648] HybridClock initialized: now 1781965900684752 us; error 39 us; skew 500 ppm
I20260620 14:31:40.686005 5079 webserver.cc:460] Webserver started at http://127.0.98.1:45763/ using document root <none> and password file <none>
I20260620 14:31:40.686245 5079 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:40.686316 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:40.686457 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:40.687486 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd3091e05fd14dc19e8cab43f4358471"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "bc7afcfabe2e9f52f521b842d20e4b6e"
server_key_iv: "cc5c9cc18172c508f47dac394b70c958"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.687855 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd3091e05fd14dc19e8cab43f4358471"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "bc7afcfabe2e9f52f521b842d20e4b6e"
server_key_iv: "cc5c9cc18172c508f47dac394b70c958"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.689132 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:31:40.689945 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.690120 5079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.690212 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "bd3091e05fd14dc19e8cab43f4358471"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "bc7afcfabe2e9f52f521b842d20e4b6e"
server_key_iv: "cc5c9cc18172c508f47dac394b70c958"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.690291 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.693588 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.693861 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.694005 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.694211 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:40.694518 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:40.694571 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.694612 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:40.694639 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.700268 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44767
I20260620 14:31:40.700340 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44767 every 8 connection(s)
I20260620 14:31:40.700678 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:40.703697 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.701410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47319 (local address 127.0.98.62:35223)
0620 14:31:40.701501 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.701504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.701791 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:31:40.702235 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.702237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.702242 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.702298 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.703494 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.703495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.703495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.703499 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.703513 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.703517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.703569 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.703629 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.703642 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:31:40.703917 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.701410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.1:47319)
0620 14:31:40.701661 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.701699 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.702140 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.702351 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.702354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.702362 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.703339 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.703340 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.703370 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.703401 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.703567 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.703569 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.703571 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.703624 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.703672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260620 14:31:40.704375 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260620 14:31:40.704486 5207 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:40.704640 5207 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260620 14:31:40.705209 5019 ts_manager.cc:194] Registered new tserver with Master: bd3091e05fd14dc19e8cab43f4358471 (127.0.98.1:44767)
I20260620 14:31:40.708348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5079
I20260620 14:31:40.708467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:40.708660 392 external_mini_cluster.cc:1428] Setting key 9650d6d09404b578df0b9268f8246144
I20260620 14:31:40.709642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--builtin_ntp_servers=127.0.98.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:31:40.789860 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:40.790064 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:40.790088 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:40.791203 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:40.791239 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:40.791263 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:40.792500 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:40.792754 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:40.793009 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:40.797855 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.797878 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.797915 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.798085 5210 server_base.cc:1034] running on GCE node
I20260620 14:31:40.798288 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:40.798568 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:40.799727 5210 hybrid_clock.cc:648] HybridClock initialized: now 1781965900799723 us; error 36 us; skew 500 ppm
I20260620 14:31:40.800871 5210 webserver.cc:460] Webserver started at http://127.0.98.2:45861/ using document root <none> and password file <none>
I20260620 14:31:40.801124 5210 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:40.801193 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:40.801327 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:40.802392 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "2df87d9363654b8182f371f9a322f553"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "579f4d8464d6597cd64ef7ca36276416"
server_key_iv: "09c41e484cefd4fd82172eac3ff2c4e2"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.802752 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2df87d9363654b8182f371f9a322f553"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "579f4d8464d6597cd64ef7ca36276416"
server_key_iv: "09c41e484cefd4fd82172eac3ff2c4e2"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.804122 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:40.804867 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.805042 5210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:40.805137 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "2df87d9363654b8182f371f9a322f553"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "579f4d8464d6597cd64ef7ca36276416"
server_key_iv: "09c41e484cefd4fd82172eac3ff2c4e2"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.805220 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.808408 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.808677 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.808820 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.809018 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:40.809311 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:40.809365 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.809404 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:40.809453 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.815071 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44749
I20260620 14:31:40.815138 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44749 every 8 connection(s)
I20260620 14:31:40.815445 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:40.818378 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.816137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35691 (local address 127.0.98.62:35223)
0620 14:31:40.816223 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.816225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.816416 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:31:40.816922 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.816923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.816930 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.816982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.818106 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.818108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.818109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.818114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.818129 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.818133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.818187 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.818291 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.818308 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:40.818552 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.816110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.2:35691)
0620 14:31:40.816306 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.816336 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.816809 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.817045 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.817051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.817064 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.817960 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.817960 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.817986 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.818012 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.818188 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.818193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.818195 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.818280 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.818341 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260620 14:31:40.819069 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260620 14:31:40.819155 5338 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:40.819269 5338 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260620 14:31:40.819591 5019 ts_manager.cc:194] Registered new tserver with Master: 2df87d9363654b8182f371f9a322f553 (127.0.98.2:44749)
I20260620 14:31:40.823943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5210
I20260620 14:31:40.824051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:40.824234 392 external_mini_cluster.cc:1428] Setting key 7db567ae4efc7356fc64dde01c0d4e3c
I20260620 14:31:40.825095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--builtin_ntp_servers=127.0.98.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:31:40.904757 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:40.904966 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:40.904989 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:40.906164 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:40.906204 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:40.906267 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:40.907678 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:40.907933 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:40.908236 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:40.913172 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:40.913172 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.913216 5341 server_base.cc:1034] running on GCE node
W20260620 14:31:40.913213 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:40.913642 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:40.913892 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:40.915081 5341 hybrid_clock.cc:648] HybridClock initialized: now 1781965900915056 us; error 35 us; skew 500 ppm
I20260620 14:31:40.916394 5341 webserver.cc:460] Webserver started at http://127.0.98.3:36703/ using document root <none> and password file <none>
I20260620 14:31:40.916635 5341 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:40.916718 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:40.916872 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:40.917982 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ed78137981b4e36bfa02ea42bd2359f"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "b07ae1ff44f8d7663817941526958546"
server_key_iv: "18b072ca979baaf194a460c82ac8f45d"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.918337 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ed78137981b4e36bfa02ea42bd2359f"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "b07ae1ff44f8d7663817941526958546"
server_key_iv: "18b072ca979baaf194a460c82ac8f45d"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.919625 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:40.920334 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.920492 5341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:40.920591 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "1ed78137981b4e36bfa02ea42bd2359f"
format_stamp: "Formatted at 2026-06-20 14:31:40 on dist-test-slave-tc2g"
server_key: "b07ae1ff44f8d7663817941526958546"
server_key_iv: "18b072ca979baaf194a460c82ac8f45d"
server_key_version: "encryptionkey@0"
I20260620 14:31:40.920679 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:40.924026 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:40.924276 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:40.924419 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:40.924616 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:40.924925 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:40.924979 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.925019 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:40.925045 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:40.930760 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45291
I20260620 14:31:40.930871 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45291 every 8 connection(s)
I20260620 14:31:40.931173 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:40.934263 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.931905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41815 (local address 127.0.98.62:35223)
0620 14:31:40.931970 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.931971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.932184 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:31:40.932687 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.932688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.932695 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.932744 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.933995 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.933997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.933997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.934002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.934017 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.934021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.934068 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.934166 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.934180 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:31:40.934448 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.931891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.98.3:41815)
0620 14:31:40.932072 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.932103 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.932565 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.932810 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.932814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.932823 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.933846 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.933846 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.933873 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.933900 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.934076 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.934081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.934083 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.934177 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.934243 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260620 14:31:40.934928 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35223
I20260620 14:31:40.935017 5469 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:40.935158 5469 heartbeater.cc:511] Master 127.0.98.62:35223 requested a full tablet report, sending...
I20260620 14:31:40.935535 5019 ts_manager.cc:194] Registered new tserver with Master: 1ed78137981b4e36bfa02ea42bd2359f (127.0.98.3:45291)
I20260620 14:31:40.939445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5341
I20260620 14:31:40.939527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:40.939658 392 external_mini_cluster.cc:1428] Setting key 9a50cbd56ed2fd4c123dbe3f0cbfaf6c
I20260620 14:31:40.940526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:40.942864 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.941853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.0.98.62:35223)
0620 14:31:40.941910 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.941912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.942008 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:31:40.942070 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.942070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.942074 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.942124 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.942721 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.942722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.942722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.942724 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.942734 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.942736 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.942771 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.942810 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.942820 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:31:40.944255 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:40.945770 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:40.949759 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.948024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.0.1:53628)
0620 14:31:40.948135 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.948144 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.948176 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.948790 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.948791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.948794 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.949332 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.949333 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.949339 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.949350 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.949599 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.949603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.949605 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.949651 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.949665 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260620 14:31:40.949802 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.947895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.0.98.2:44749)
0620 14:31:40.948059 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.948064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.948074 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:40.948098 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.948100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.948114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.948195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.949447 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.949455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.949456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.949461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.949494 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.949539 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.949605 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.949692 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.949709 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 14:31:40.949939 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.948437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.0.98.3:45291)
0620 14:31:40.948622 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.948627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.948637 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:40.948663 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.948666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.948674 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.948972 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.949566 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.949573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.949573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.949577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.949605 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.949627 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.949813 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.949835 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.949850 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260620 14:31:40.949934 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.948456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.0.98.1:44767)
0620 14:31:40.948642 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:40.948647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:40.948656 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:40.948679 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:40.948681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:40.948688 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:40.948972 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:40.949451 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:40.949457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:40.949457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:40.949461 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:40.949495 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:40.949519 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:40.949799 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:40.949823 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:31:40.949839 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260620 14:31:40.950405 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.948195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45291 (local address 127.0.0.1:43348)
0620 14:31:40.948298 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.948307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.948340 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.948792 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.948793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.948794 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.949462 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.949463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.949470 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.949481 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.949711 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.949711 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.949712 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.949750 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.949762 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260620 14:31:40.949759 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:40.947803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44749 (local address 127.0.0.1:35934)
0620 14:31:40.947921 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:40.947959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:40.948008 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:40.948214 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:40.948219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:40.948226 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:40.948825 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:40.948826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:40.948844 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:40.948868 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:40.949596 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:40.949603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:40.949605 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:40.949678 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:31:40.949688 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":76,"threads_started":1}
I20260620 14:31:40.950639 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f (DEFAULT_TABLE table=test-table [id=e9ccb1e7782648b2b1bf318f508b8ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.951062 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8ca29c7b914db09176a6c64b08fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.951045 5403 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f (DEFAULT_TABLE table=test-table [id=e9ccb1e7782648b2b1bf318f508b8ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.951363 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8ca29c7b914db09176a6c64b08fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.951352 5272 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f (DEFAULT_TABLE table=test-table [id=e9ccb1e7782648b2b1bf318f508b8ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:40.951632 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8ca29c7b914db09176a6c64b08fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:40.953841 5483 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Bootstrap starting.
I20260620 14:31:40.954495 5483 tablet_bootstrap.cc:654] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.954592 5484 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: Bootstrap starting.
I20260620 14:31:40.954929 5485 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Bootstrap starting.
I20260620 14:31:40.954944 5483 log.cc:824] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.955252 5484 tablet_bootstrap.cc:654] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.955634 5484 log.cc:824] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.955703 5485 tablet_bootstrap.cc:654] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:40.956000 5485 log.cc:824] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:40.956100 5483 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: No bootstrap required, opened a new log
I20260620 14:31:40.956172 5483 ts_tablet_manager.cc:1397] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:40.956375 5484 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: No bootstrap required, opened a new log
I20260620 14:31:40.956449 5484 ts_tablet_manager.cc:1397] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:40.956664 5485 tablet_bootstrap.cc:492] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: No bootstrap required, opened a new log
I20260620 14:31:40.956712 5485 ts_tablet_manager.cc:1397] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:40.958541 5485 raft_consensus.cc:348] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.958736 5485 raft_consensus.cc:374] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.958776 5485 raft_consensus.cc:729] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df87d9363654b8182f371f9a322f553, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.958889 5485 consensus_queue.cc:260] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.958825 5483 raft_consensus.cc:348] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.958992 5483 raft_consensus.cc:374] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.958935 5484 raft_consensus.cc:348] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.959152 5484 raft_consensus.cc:374] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:40.959162 5485 ts_tablet_manager.cc:1428] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:40.959203 5484 raft_consensus.cc:729] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3091e05fd14dc19e8cab43f4358471, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.959282 5338 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260620 14:31:40.959033 5483 raft_consensus.cc:729] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed78137981b4e36bfa02ea42bd2359f, State: Initialized, Role: FOLLOWER
I20260620 14:31:40.959335 5484 consensus_queue.cc:260] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.959415 5483 consensus_queue.cc:260] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:40.959601 5484 ts_tablet_manager.cc:1428] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:40.959643 5483 ts_tablet_manager.cc:1428] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:31:40.959719 5207 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
I20260620 14:31:40.959745 5469 heartbeater.cc:503] Master 127.0.98.62:35223 was elected leader, sending a full tablet report...
W20260620 14:31:41.065958 5339 tablet.cc:2369] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:41.117564 5490 raft_consensus.cc:482] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:41.117683 5490 raft_consensus.cc:504] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:41.117977 5490 leader_election.cc:288] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df87d9363654b8182f371f9a322f553 (127.0.98.2:44749), 1ed78137981b4e36bfa02ea42bd2359f (127.0.98.3:45291)
I20260620 14:31:41.119448 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.118288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53885 (local address 127.0.98.2:44749)
0620 14:31:41.118344 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.118346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.118391 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:31:41.118417 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.118418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.118422 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.118493 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.119163 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.119165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.119165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.119170 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.119193 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.119197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.119274 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.119371 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.119387 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:41.119448 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.118199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44749 (local address 127.0.98.1:53885)
0620 14:31:41.118265 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.118277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:41.118322 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:41.118552 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:41.118553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:41.118556 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:41.119054 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:41.119055 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:41.119063 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:41.119076 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:41.119280 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:41.119282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:41.119282 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:31:41.119363 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:31:41.119378 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:31:41.119611 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.118495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43705 (local address 127.0.98.3:45291)
0620 14:31:41.118565 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.118567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.118577 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:41.118642 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.118643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.118648 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.118722 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.119351 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.119352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.119353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.119357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.119369 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.119373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.119447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.119501 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.119515 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:41.119788 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.118334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45291 (local address 127.0.98.1:43705)
0620 14:31:41.118507 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.118535 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:41.118580 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:41.118727 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:41.118727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:41.118730 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:41.119256 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:41.119257 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:41.119264 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:41.119277 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:41.119444 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:41.119444 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:41.119445 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:31:41.119619 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:31:41.119632 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
I20260620 14:31:41.119800 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8ca29c7b914db09176a6c64b08fd8f" candidate_uuid: "bd3091e05fd14dc19e8cab43f4358471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df87d9363654b8182f371f9a322f553" is_pre_election: true
I20260620 14:31:41.119961 5292 raft_consensus.cc:2454] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3091e05fd14dc19e8cab43f4358471 in term 0.
I20260620 14:31:41.120085 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8ca29c7b914db09176a6c64b08fd8f" candidate_uuid: "bd3091e05fd14dc19e8cab43f4358471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed78137981b4e36bfa02ea42bd2359f" is_pre_election: true
I20260620 14:31:41.120182 5097 leader_election.cc:302] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df87d9363654b8182f371f9a322f553, bd3091e05fd14dc19e8cab43f4358471; no voters:
I20260620 14:31:41.120239 5423 raft_consensus.cc:2454] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3091e05fd14dc19e8cab43f4358471 in term 0.
I20260620 14:31:41.120325 5490 raft_consensus.cc:2781] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:41.120399 5490 raft_consensus.cc:482] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:41.120440 5490 raft_consensus.cc:3037] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.121133 5490 raft_consensus.cc:504] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:41.121258 5490 leader_election.cc:288] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [CANDIDATE]: Term 1 election: Requested vote from peers 2df87d9363654b8182f371f9a322f553 (127.0.98.2:44749), 1ed78137981b4e36bfa02ea42bd2359f (127.0.98.3:45291)
I20260620 14:31:41.121394 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8ca29c7b914db09176a6c64b08fd8f" candidate_uuid: "bd3091e05fd14dc19e8cab43f4358471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df87d9363654b8182f371f9a322f553"
I20260620 14:31:41.121438 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8ca29c7b914db09176a6c64b08fd8f" candidate_uuid: "bd3091e05fd14dc19e8cab43f4358471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed78137981b4e36bfa02ea42bd2359f"
I20260620 14:31:41.121491 5292 raft_consensus.cc:3037] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.121526 5423 raft_consensus.cc:3037] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.122166 5292 raft_consensus.cc:2454] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3091e05fd14dc19e8cab43f4358471 in term 1.
I20260620 14:31:41.122244 5423 raft_consensus.cc:2454] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3091e05fd14dc19e8cab43f4358471 in term 1.
I20260620 14:31:41.122385 5097 leader_election.cc:302] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df87d9363654b8182f371f9a322f553, bd3091e05fd14dc19e8cab43f4358471; no voters:
I20260620 14:31:41.122490 5490 raft_consensus.cc:2781] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:41.122701 5490 raft_consensus.cc:686] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [term 1 LEADER]: Becoming Leader. State: Replica: bd3091e05fd14dc19e8cab43f4358471, State: Running, Role: LEADER
I20260620 14:31:41.122808 5490 consensus_queue.cc:237] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } }
I20260620 14:31:41.123667 5018 catalog_manager.cc:5495] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd3091e05fd14dc19e8cab43f4358471 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd3091e05fd14dc19e8cab43f4358471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd3091e05fd14dc19e8cab43f4358471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:41.131314 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.130373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.0.98.1:44767)
0620 14:31:41.130414 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.130416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.130425 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:41.130501 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.130502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.130514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.130587 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.131087 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.131088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.131089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.131093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.131106 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.131110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.131152 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.131236 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.131248 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:31:41.132905 5423 raft_consensus.cc:1261] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 1 FOLLOWER]: Refusing update from remote peer bd3091e05fd14dc19e8cab43f4358471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:41.132915 5292 raft_consensus.cc:1261] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 1 FOLLOWER]: Refusing update from remote peer bd3091e05fd14dc19e8cab43f4358471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:41.133232 5493 consensus_queue.cc:1035] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed78137981b4e36bfa02ea42bd2359f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:41.133348 5490 consensus_queue.cc:1035] T 5a8ca29c7b914db09176a6c64b08fd8f P bd3091e05fd14dc19e8cab43f4358471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df87d9363654b8182f371f9a322f553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:41.135548 5496 mvcc.cc:205] Tried to move back new op lower bound from 7298932331037822976 to 7298932330999607296. Current Snapshot: MvccSnapshot[applied={T|T < 7298932331037822976}]
I20260620 14:31:41.135785 5498 mvcc.cc:205] Tried to move back new op lower bound from 7298932331037822976 to 7298932330999607296. Current Snapshot: MvccSnapshot[applied={T|T < 7298932331037822976}]
I20260620 14:31:41.138265 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:41.138371 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:41.139066 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 010a6cab0c7c4a63a114dda46d3573db: Sending DeleteTablet for 3 replicas of tablet 5a8ca29c7b914db09176a6c64b08fd8f
I20260620 14:31:41.139408 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:41 UTC) from {username='slave'} at 127.0.0.1:35934
I20260620 14:31:41.139459 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:41 UTC) from {username='slave'} at 127.0.0.1:43348
I20260620 14:31:41.139655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5079
I20260620 14:31:41.139635 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8ca29c7b914db09176a6c64b08fd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:41 UTC) from {username='slave'} at 127.0.0.1:53628
W20260620 14:31:41.145223 5006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44767 (error 108)
I20260620 14:31:41.145293 5507 tablet_replica.cc:330] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: stopping tablet replica
I20260620 14:31:41.145293 5506 tablet_replica.cc:330] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553: stopping tablet replica
I20260620 14:31:41.145401 5506 raft_consensus.cc:2229] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:41.145401 5507 raft_consensus.cc:2229] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:41.145532 5507 raft_consensus.cc:2258] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:41.145532 5506 raft_consensus.cc:2258] T 5a8ca29c7b914db09176a6c64b08fd8f P 2df87d9363654b8182f371f9a322f553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:41.145776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5210
I20260620 14:31:41.145792 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.145633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.0.1:53644)
0620 14:31:41.145704 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.145753 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:31:41.145931 5006 catalog_manager.cc:4584] TS bd3091e05fd14dc19e8cab43f4358471 (127.0.98.1:44767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a8ca29c7b914db09176a6c64b08fd8f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44767: connect: Connection refused (error 111)
I20260620 14:31:41.146641 5507 ts_tablet_manager.cc:1905] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:41.148648 5507 ts_tablet_manager.cc:1918] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:41.148725 5507 log.cc:1196] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/5a8ca29c7b914db09176a6c64b08fd8f
I20260620 14:31:41.148970 5507 ts_tablet_manager.cc:1939] T 5a8ca29c7b914db09176a6c64b08fd8f P 1ed78137981b4e36bfa02ea42bd2359f: Deleting consensus metadata
I20260620 14:31:41.149391 5006 catalog_manager.cc:4841] TS 1ed78137981b4e36bfa02ea42bd2359f (127.0.98.3:45291): tablet 5a8ca29c7b914db09176a6c64b08fd8f (table test-table [id=e9ccb1e7782648b2b1bf318f508b8ed0]) successfully deleted
I20260620 14:31:41.151213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5341
I20260620 14:31:41.151270 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.151135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44749 (local address 127.0.0.1:35942)
0620 14:31:41.151220 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.151246 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:31:41.155848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 4988
2026-06-20T14:31:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (740 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:31:41.175920 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:31:41.176992 392 webserver.cc:460] Webserver started at https://127.0.0.1:46437/ using document root <none> and password file <none>
I20260620 14:31:41.177120 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:31:41.177302 392 webserver.cc:460] Webserver started at http://127.0.0.1:42527/ using document root <none> and password file <none>
2026-06-20T14:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:41Z Disabled control of system clock
I20260620 14:31:41.184607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35459
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46437/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/testchainca.cert with env {}
W20260620 14:31:41.265312 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:41.265522 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:41.265544 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:41.266702 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:41.266741 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:41.266753 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:41.266765 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:41.266776 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:31:41.266794 5514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:31:41.266805 5514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46437/jwks.json
I20260620 14:31:41.268275 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46437/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:41.268522 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:41.268757 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:41.273281 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.273289 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:41.273406 5514 server_base.cc:1034] running on GCE node
W20260620 14:31:41.273289 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:41.273778 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:41.274125 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:41.275269 5514 hybrid_clock.cc:648] HybridClock initialized: now 1781965901275253 us; error 35 us; skew 500 ppm
I20260620 14:31:41.276580 5514 webserver.cc:460] Webserver started at http://127.0.98.62:41379/ using document root <none> and password file <none>
I20260620 14:31:41.276887 5514 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:41.276939 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:41.277086 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:41.278249 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "85227933eab549658429d72432747f51"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "c431464a1bccb8f9569ee2f30db4c4cf"
server_key_iv: "05e450624aa16a3d30b061e59574efe3"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.278613 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "85227933eab549658429d72432747f51"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "c431464a1bccb8f9569ee2f30db4c4cf"
server_key_iv: "05e450624aa16a3d30b061e59574efe3"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.279850 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:41.280514 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:41.280682 5514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:41.280772 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "85227933eab549658429d72432747f51"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "c431464a1bccb8f9569ee2f30db4c4cf"
server_key_iv: "05e450624aa16a3d30b061e59574efe3"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.280857 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:41.290992 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:41.291278 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:41.291426 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:41.295346 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35459
I20260620 14:31:41.295375 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35459 every 8 connection(s)
I20260620 14:31:41.295742 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:41.296655 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:41.298900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5514
I20260620 14:31:41.299010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:41.299113 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51: Bootstrap starting.
I20260620 14:31:41.299207 392 external_mini_cluster.cc:1428] Setting key ee1b6c6031e692d37cb4c8d9279eeee5
I20260620 14:31:41.299832 5581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:41.300117 5581 log.cc:824] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:41.300872 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51: No bootstrap required, opened a new log
I20260620 14:31:41.303036 5581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260620 14:31:41.303179 5581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:41.303208 5581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85227933eab549658429d72432747f51, State: Initialized, Role: FOLLOWER
I20260620 14:31:41.303356 5581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260620 14:31:41.303414 5581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:41.303434 5581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:41.303498 5581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.304174 5581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260620 14:31:41.304288 5581 leader_election.cc:302] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85227933eab549658429d72432747f51; no voters:
I20260620 14:31:41.304440 5581 leader_election.cc:288] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:41.304486 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:41.304644 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [term 1 LEADER]: Becoming Leader. State: Replica: 85227933eab549658429d72432747f51, State: Running, Role: LEADER
I20260620 14:31:41.304718 5581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:41.305060 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.299680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.0.98.62:35459)
0620 14:31:41.299880 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.299889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.299922 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:31:41.300422 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.300430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.300506 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.300640 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.300864 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.301862 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.303032 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.303606 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.303690 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.304335 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.304339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.304339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.304572 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.304617 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.304632 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.304695 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.304785 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.304869 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 14:31:41.304729 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } }
I20260620 14:31:41.305676 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85227933eab549658429d72432747f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260620 14:31:41.305687 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85227933eab549658429d72432747f51. Latest consensus state: current_term: 1 leader_uuid: "85227933eab549658429d72432747f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85227933eab549658429d72432747f51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35459 } } }
I20260620 14:31:41.305778 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.305794 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.306001 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:41.306026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:31:41.306461 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:41.308215 5598 catalog_manager.cc:1347] Generated new cluster ID: 0f72ba74b81a4650bc87806a5118e515
I20260620 14:31:41.308307 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:41.328686 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:41.329278 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:41.340880 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85227933eab549658429d72432747f51: Generated new TSK 0
I20260620 14:31:41.341085 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:31:41.362721 5611 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:31:41.362831 5611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:31:41.363080 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.347888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.98.62:35459)
0620 14:31:41.347933 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.347935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.348066 (+ 131us) server_negotiation.cc:407] Connection header received
0620 14:31:41.348175 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.348176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.348186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.348247 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:31:41.348426 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.348931 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.350105 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.350293 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.350345 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.350574 (+ 229us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:31:41.362855 (+ 12281us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:31:41.363030 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:31:41.363152 5584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:31:41.363528 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:41.347778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35459 (local address 127.0.0.1:39960)
0620 14:31:41.347988 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.348001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:41.348075 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:41.348271 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:41.348272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:41.348273 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:31:41.348355 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.348357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.349033 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.349034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.350027 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.350030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.350368 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.350372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.350437 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.350459 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:31:41.363041 (+ 12582us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:31:41.363199 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:31:41.364243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5514
2026-06-20T14:31:41Z chronyd exiting
I20260620 14:31:41.400058 392 test_util.cc:175] -----------------------------------------------
I20260620 14:31:41.400164 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965876904925-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:31:41.400969 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:31:41.401724 392 webserver.cc:460] Webserver started at https://127.0.0.1:38843/ using document root <none> and password file <none>
I20260620 14:31:41.401798 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:31:41.401960 392 webserver.cc:460] Webserver started at http://127.0.0.1:43739/ using document root <none> and password file <none>
2026-06-20T14:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:41Z Disabled control of system clock
I20260620 14:31:41.409650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33089
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/testchainca.cert with env {}
W20260620 14:31:41.492681 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:41.492885 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:41.492908 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:41.494127 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:41.494166 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:41.494179 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:41.494215 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:41.494241 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:31:41.494267 5617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:31:41.494294 5617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38843/jwks.json
I20260620 14:31:41.495767 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:41.496017 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:41.496248 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:41.500921 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.500931 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.500947 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:41.501142 5617 server_base.cc:1034] running on GCE node
I20260620 14:31:41.501374 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:41.501706 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:41.502861 5617 hybrid_clock.cc:648] HybridClock initialized: now 1781965901502841 us; error 37 us; skew 500 ppm
I20260620 14:31:41.504087 5617 webserver.cc:460] Webserver started at http://127.0.98.62:43463/ using document root <none> and password file <none>
I20260620 14:31:41.504330 5617 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:41.504379 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:41.504503 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:41.505585 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "fd3ad53a879c49a9886480924bf4e1a9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "9478172cb56f12166b612efa91596f8c"
server_key_iv: "52c70f3c78eb3d1fed4c7f0e27d52639"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.505941 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd3ad53a879c49a9886480924bf4e1a9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "9478172cb56f12166b612efa91596f8c"
server_key_iv: "52c70f3c78eb3d1fed4c7f0e27d52639"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.507212 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:31:41.507889 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:41.508059 5617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:41.508141 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "fd3ad53a879c49a9886480924bf4e1a9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "9478172cb56f12166b612efa91596f8c"
server_key_iv: "52c70f3c78eb3d1fed4c7f0e27d52639"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.508198 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:41.521836 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:41.522076 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:41.522181 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:41.525929 5617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33089
I20260620 14:31:41.525998 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33089 every 8 connection(s)
I20260620 14:31:41.526296 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:41.527221 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:41.529397 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9: Bootstrap starting.
I20260620 14:31:41.530027 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:41.530303 5684 log.cc:824] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:41.530998 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9: No bootstrap required, opened a new log
I20260620 14:31:41.532544 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260620 14:31:41.532697 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:41.532757 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3ad53a879c49a9886480924bf4e1a9, State: Initialized, Role: FOLLOWER
I20260620 14:31:41.532940 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260620 14:31:41.533038 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:41.533083 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:41.533135 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.533707 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260620 14:31:41.533828 5684 leader_election.cc:302] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd3ad53a879c49a9886480924bf4e1a9; no voters:
I20260620 14:31:41.533854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5617
I20260620 14:31:41.533973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:41.534024 5684 leader_election.cc:288] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:41.534051 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:41.534170 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [term 1 LEADER]: Becoming Leader. State: Replica: fd3ad53a879c49a9886480924bf4e1a9, State: Running, Role: LEADER
I20260620 14:31:41.534197 392 external_mini_cluster.cc:1428] Setting key be523d069f45383c414b04d0bb7345a6
I20260620 14:31:41.534277 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:41.534263 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260620 14:31:41.534584 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd3ad53a879c49a9886480924bf4e1a9. Latest consensus state: current_term: 1 leader_uuid: "fd3ad53a879c49a9886480924bf4e1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } }
I20260620 14:31:41.534677 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd3ad53a879c49a9886480924bf4e1a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3ad53a879c49a9886480924bf4e1a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } }
I20260620 14:31:41.534767 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.534735 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.535053 5696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:41.535614 5696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:41.537505 5696 catalog_manager.cc:1347] Generated new cluster ID: e4df4354dbf34133b325f4a9cf7dda5d
I20260620 14:31:41.537559 5696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:41.541240 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.534821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.98.62:33089)
0620 14:31:41.535596 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.535604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.535636 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:31:41.535753 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.535760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.535834 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.536456 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.536716 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.538030 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.539090 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.539720 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.539843 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.540536 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.540540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.540540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.540745 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.540791 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.540805 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.540875 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.540942 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.541017 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":87,"threads_started":1}
I20260620 14:31:41.542131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:31:41.548832 5696 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:41.549536 5696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:41.557569 5696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd3ad53a879c49a9886480924bf4e1a9: Generated new TSK 0
I20260620 14:31:41.557765 5696 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:31:41.579352 5714 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:31:41.579445 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:31:41.579743 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.565250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.98.62:33089)
0620 14:31:41.565295 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.565297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.565397 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:31:41.565460 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.565461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.565473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.565556 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:31:41.565687 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.566561 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.567325 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.567648 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.567718 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.567876 (+ 158us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:31:41.579518 (+ 11642us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:31:41.579671 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:31:41.579802 5697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:31:41.579818 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:41.565172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33089 (local address 127.0.0.1:35146)
0620 14:31:41.565327 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.565337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:41.565390 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:41.565560 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:41.565561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:41.565563 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:31:41.565609 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.565611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.566652 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.566653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.567237 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.567238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.567724 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.567727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.567775 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.567780 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:31:41.579632 (+ 11852us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:31:41.579723 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33089: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260620 14:31:41.580399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5617
2026-06-20T14:31:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:31:41.618073 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:31:41.618779 392 webserver.cc:460] Webserver started at https://127.0.0.1:43781/ using document root <none> and password file <none>
I20260620 14:31:41.618824 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:31:41.619001 392 webserver.cc:460] Webserver started at http://127.0.0.1:33281/ using document root <none> and password file <none>
2026-06-20T14:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:41Z Disabled control of system clock
I20260620 14:31:41.626581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43781/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:31:41.704880 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:41.705129 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:41.705168 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:41.706413 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:41.706509 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:41.706532 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:41.706552 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:41.706590 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:31:41.706616 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:31:41.706641 5720 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43781/jwks.json
I20260620 14:31:41.708073 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43781/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:41.708366 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:41.708679 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:41.713336 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.713351 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.713357 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:41.713590 5720 server_base.cc:1034] running on GCE node
I20260620 14:31:41.713835 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:41.714124 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:41.715294 5720 hybrid_clock.cc:648] HybridClock initialized: now 1781965901715282 us; error 29 us; skew 500 ppm
I20260620 14:31:41.716462 5720 webserver.cc:460] Webserver started at http://127.0.98.62:41901/ using document root <none> and password file <none>
I20260620 14:31:41.716691 5720 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:41.716763 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:41.716915 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:41.718067 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "e9ef7b18358941059b4f42dce4085fe9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "4e516eaa134a1d59314c27e8b6674739"
server_key_iv: "0ae2f401349587ce53e958b7bc9015dc"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.718462 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9ef7b18358941059b4f42dce4085fe9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "4e516eaa134a1d59314c27e8b6674739"
server_key_iv: "0ae2f401349587ce53e958b7bc9015dc"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.719774 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:41.720559 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:41.720728 5720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:41.720830 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "e9ef7b18358941059b4f42dce4085fe9"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "4e516eaa134a1d59314c27e8b6674739"
server_key_iv: "0ae2f401349587ce53e958b7bc9015dc"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.720918 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:41.764338 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:41.764701 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:41.764863 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:41.768671 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44379
I20260620 14:31:41.768716 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44379 every 8 connection(s)
I20260620 14:31:41.769086 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:41.769816 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:41.770826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5720
I20260620 14:31:41.770934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:41.771114 392 external_mini_cluster.cc:1428] Setting key 647b4480396037731b660dc29c4d6d13
I20260620 14:31:41.773312 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9: Bootstrap starting.
I20260620 14:31:41.774021 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:41.774319 5787 log.cc:824] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:41.775058 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9: No bootstrap required, opened a new log
I20260620 14:31:41.777110 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260620 14:31:41.777297 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.771638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.98.62:44379)
0620 14:31:41.771872 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.771881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.771913 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:31:41.772446 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.772457 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.772541 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.772699 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.772861 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.774208 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.775006 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.775600 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.775725 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.776351 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.776356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.776356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.776559 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.776605 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.776621 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.776688 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.776743 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.776819 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260620 14:31:41.777300 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:41.777369 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ef7b18358941059b4f42dce4085fe9, State: Initialized, Role: FOLLOWER
I20260620 14:31:41.777513 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260620 14:31:41.777592 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:41.777616 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:41.777680 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.778335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:31:41.778326 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260620 14:31:41.778431 5787 leader_election.cc:302] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9ef7b18358941059b4f42dce4085fe9; no voters:
I20260620 14:31:41.778578 5787 leader_election.cc:288] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:41.778633 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:41.778782 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [term 1 LEADER]: Becoming Leader. State: Replica: e9ef7b18358941059b4f42dce4085fe9, State: Running, Role: LEADER
I20260620 14:31:41.778842 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:41.778865 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260620 14:31:41.779168 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9ef7b18358941059b4f42dce4085fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260620 14:31:41.779268 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.779451 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ef7b18358941059b4f42dce4085fe9. Latest consensus state: current_term: 1 leader_uuid: "e9ef7b18358941059b4f42dce4085fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ef7b18358941059b4f42dce4085fe9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260620 14:31:41.779516 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.779500 5798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:41.780054 5798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:41.781608 5798 catalog_manager.cc:1347] Generated new cluster ID: 9f269a328ea34c02971338d71a351363
I20260620 14:31:41.781659 5798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:41.795209 5798 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:41.795895 5798 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:41.805227 5798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9ef7b18358941059b4f42dce4085fe9: Generated new TSK 0
I20260620 14:31:41.805449 5798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:41.815261 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.811888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.0.98.62:44379)
0620 14:31:41.811940 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.811944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.812050 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:31:41.812135 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.812136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.812148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.812240 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:31:41.812361 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.812876 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.813742 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.813944 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.814019 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.814204 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:31:41.815016 (+ 812us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:31:41.815186 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:31:41.815320 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:31:41.815347 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:41.811831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.0.1:59544)
0620 14:31:41.811976 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:41.811987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:41.812068 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:41.812242 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:41.812243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:41.812245 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:31:41.812298 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.812300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.812973 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.812974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.813650 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:41.813652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.814016 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.814024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:41.814088 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.814093 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:31:41.815149 (+ 1056us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:31:41.815238 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260620 14:31:41.815654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5720
E20260620 14:31:41.826089 5817 webserver.cc:533] Webserver: sslize failed:
E20260620 14:31:41.826157 5817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:31:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:41Z Disabled control of system clock
I20260620 14:31:41.860666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37305
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37305
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/testchainca.cert with env {}
W20260620 14:31:41.940795 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:41.940994 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:41.941033 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:41.942178 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:41.942215 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:41.942229 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:41.942241 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:41.942253 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:31:41.942266 5821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:31:41.942277 5821 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:31:41.943674 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:41.943861 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:41.944013 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:41.948925 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.948933 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:41.948937 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:41.949020 5821 server_base.cc:1034] running on GCE node
I20260620 14:31:41.949380 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:41.949744 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:41.950918 5821 hybrid_clock.cc:648] HybridClock initialized: now 1781965901950901 us; error 29 us; skew 500 ppm
I20260620 14:31:41.952080 5821 webserver.cc:460] Webserver started at http://127.0.98.62:38721/ using document root <none> and password file <none>
I20260620 14:31:41.952320 5821 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:41.952389 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:41.952535 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:41.953688 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "3460132622e446ce9b57c879c59a29a7"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "b24612690626f025160aae4e1b970121"
server_key_iv: "2eae311cb2c4e29dc1dfac6a3475d886"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.954124 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "3460132622e446ce9b57c879c59a29a7"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "b24612690626f025160aae4e1b970121"
server_key_iv: "2eae311cb2c4e29dc1dfac6a3475d886"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.955360 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:41.956075 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:41.956298 5821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:41.956399 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "3460132622e446ce9b57c879c59a29a7"
format_stamp: "Formatted at 2026-06-20 14:31:41 on dist-test-slave-tc2g"
server_key: "b24612690626f025160aae4e1b970121"
server_key_iv: "2eae311cb2c4e29dc1dfac6a3475d886"
server_key_version: "encryptionkey@0"
I20260620 14:31:41.956470 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:41.974711 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:41.975047 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:41.975205 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:41.979116 5821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37305
I20260620 14:31:41.979172 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37305 every 8 connection(s)
I20260620 14:31:41.979527 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:41.980501 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:41.982849 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7: Bootstrap starting.
I20260620 14:31:41.983415 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:41.983700 5888 log.cc:824] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:41.984336 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7: No bootstrap required, opened a new log
I20260620 14:31:41.984992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5821
I20260620 14:31:41.985087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:41.985293 392 external_mini_cluster.cc:1428] Setting key 986c38432c0cda0f3c20846431bd2b0b
I20260620 14:31:41.986567 5888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } }
I20260620 14:31:41.986752 5888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:41.986795 5888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3460132622e446ce9b57c879c59a29a7, State: Initialized, Role: FOLLOWER
I20260620 14:31:41.986944 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } }
I20260620 14:31:41.987035 5888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:41.987078 5888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:41.987129 5888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:41.987643 5888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } }
I20260620 14:31:41.987757 5888 leader_election.cc:302] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3460132622e446ce9b57c879c59a29a7; no voters:
I20260620 14:31:41.987941 5888 leader_election.cc:288] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:41.987968 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:41.988072 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3460132622e446ce9b57c879c59a29a7, State: Running, Role: LEADER
I20260620 14:31:41.988162 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } }
I20260620 14:31:41.988201 5888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:41.988464 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3460132622e446ce9b57c879c59a29a7. Latest consensus state: current_term: 1 leader_uuid: "3460132622e446ce9b57c879c59a29a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } } }
I20260620 14:31:41.988557 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.988762 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3460132622e446ce9b57c879c59a29a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3460132622e446ce9b57c879c59a29a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37305 } } }
I20260620 14:31:41.988838 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:41.988793 5898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:41.989341 5898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:41.991530 5898 catalog_manager.cc:1347] Generated new cluster ID: 9c2e9dafac6346a5805e9239aa39af61
I20260620 14:31:41.991592 5898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:41.992147 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:41.985808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.62:37305)
0620 14:31:41.986019 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:41.986035 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:41.986058 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:31:41.986497 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:41.986504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:41.986561 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:41.986754 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:41.986966 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.988434 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.989678 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:41.990310 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:41.990421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:41.991267 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:41.991271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:41.991272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:41.991481 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:41.991524 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:41.991540 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:41.991605 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:41.991877 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:31:41.991956 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 14:31:41.993105 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:31:42.004375 5898 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:42.004904 5898 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:42.009671 5898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3460132622e446ce9b57c879c59a29a7: Generated new TSK 0
I20260620 14:31:42.009858 5898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:42.100040 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:42.018729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.98.62:37305)
0620 14:31:42.018786 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:42.018789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:42.018892 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:31:42.018956 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:42.018957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:42.018969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:42.019037 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:31:42.019171 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:42.019983 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:42.020792 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:42.021109 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:42.021180 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:42.021334 (+ 154us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:31:42.099771 (+ 78437us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:31:42.099964 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53918: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:31:42.100131 5892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53918: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:31:42.100163 5917 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:42.018646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37305 (local address 127.0.0.1:53918)
0620 14:31:42.018814 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:42.018825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:42.018889 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:42.019043 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:42.019044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:42.019046 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:31:42.019109 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:42.019111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:42.020095 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:42.020096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:42.020706 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:42.020708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:42.021185 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:42.021191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:42.021243 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:42.021248 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:31:42.099930 (+ 78682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:31:42.100047 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260620 14:31:42.100504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5821
2026-06-20T14:31:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (264 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:42 dist-test-slave-tc2g krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:42 dist-test-slave-tc2g krb5kdc[5929](info): set up 2 sockets
Jun 20 14:31:42 dist-test-slave-tc2g krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260620 14:31:43.305513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:43.417687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:43.499550 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:43.499771 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:43.499794 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:43.500953 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:43.500998 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:43.501011 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:43.501024 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:43.501076 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:43.502653 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:43.502907 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:43.503124 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:43.507804 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:43.507808 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:43.507861 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:43.507995 5942 server_base.cc:1034] running on GCE node
I20260620 14:31:43.508260 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:43.508508 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:43.509675 5942 hybrid_clock.cc:648] HybridClock initialized: now 1781965903509640 us; error 47 us; skew 500 ppm
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:43.512198 5942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:43.512578 5942 webserver.cc:460] Webserver started at http://127.0.98.62:36783/ using document root <none> and password file <none>
I20260620 14:31:43.512766 5942 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:43.512806 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:43.512893 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:43.513975 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "8cd41a9e43644ef68efd9f0a838fa766"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "3a0362418b3896609c6838f86d26ef31"
server_key_iv: "2181cc744386bd5d717d5fbb77c78039"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.514272 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cd41a9e43644ef68efd9f0a838fa766"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "3a0362418b3896609c6838f86d26ef31"
server_key_iv: "2181cc744386bd5d717d5fbb77c78039"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.515578 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:43.516322 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.516492 5942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:43.516575 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "8cd41a9e43644ef68efd9f0a838fa766"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "3a0362418b3896609c6838f86d26ef31"
server_key_iv: "2181cc744386bd5d717d5fbb77c78039"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.516630 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:43.530254 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:43.532158 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:43.532287 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:43.536464 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45825
I20260620 14:31:43.536541 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45825 every 8 connection(s)
I20260620 14:31:43.537029 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:43.537736 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:43.539935 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: Bootstrap starting.
I20260620 14:31:43.540527 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:43.540824 6010 log.cc:824] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:43.541882 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: No bootstrap required, opened a new log
I20260620 14:31:43.541880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 5942
I20260620 14:31:43.541997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:43.542218 392 external_mini_cluster.cc:1428] Setting key 1029486ba112bc4ab64212d2470cc51b
I20260620 14:31:43.544770 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260620 14:31:43.544956 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:43.544991 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd41a9e43644ef68efd9f0a838fa766, State: Initialized, Role: FOLLOWER
I20260620 14:31:43.545110 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260620 14:31:43.545192 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:43.545222 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:43.545259 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:43.545982 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260620 14:31:43.546069 6010 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd41a9e43644ef68efd9f0a838fa766; no voters:
I20260620 14:31:43.546245 6010 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:43.546273 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:43.546379 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd41a9e43644ef68efd9f0a838fa766, State: Running, Role: LEADER
I20260620 14:31:43.546501 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:43.546510 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } }
I20260620 14:31:43.546963 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd41a9e43644ef68efd9f0a838fa766. Latest consensus state: current_term: 1 leader_uuid: "8cd41a9e43644ef68efd9f0a838fa766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } } }
I20260620 14:31:43.547089 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:43.547281 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd41a9e43644ef68efd9f0a838fa766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd41a9e43644ef68efd9f0a838fa766" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45825 } } }
I20260620 14:31:43.547382 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766 [sys.catalog]: This master's current role is: LEADER
W20260620 14:31:43.548349 6031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:31:43.548413 6031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:31:43.548502 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:43.548710 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:43.550154 6024 catalog_manager.cc:1347] Generated new cluster ID: 5d42eec1ad0349848d06c9c075c474f1
I20260620 14:31:43.550201 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:43.554288 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:43.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.98.62:45825)
0620 14:31:43.542862 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:43.542872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:43.542923 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:31:43.543371 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:43.543380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:43.543441 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:43.543592 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:43.544243 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.545745 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.546743 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.547397 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.547485 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:43.550548 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:43.550552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:43.550552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:43.550563 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:43.552528 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.552974 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.552978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.552978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.553083 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.553294 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.553295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.553295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.553647 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:43.553847 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:43.553948 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:31:43.554082 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260620 14:31:43.571712 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:43.572453 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:43.580446 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd41a9e43644ef68efd9f0a838fa766: Generated new TSK 0
I20260620 14:31:43.580688 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:43.612943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:43.698071 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:43.698307 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:43.698343 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:43.699491 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:43.699555 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:43.699599 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:43.701035 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:43.701318 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:43.701562 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:43.706444 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:43.706524 6036 server_base.cc:1034] running on GCE node
W20260620 14:31:43.706482 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:43.706456 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:43.706861 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:43.707090 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:43.708240 6036 hybrid_clock.cc:648] HybridClock initialized: now 1781965903708227 us; error 43 us; skew 500 ppm
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:43.710753 6036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:43.711205 6036 webserver.cc:460] Webserver started at http://127.0.98.1:44841/ using document root <none> and password file <none>
I20260620 14:31:43.711444 6036 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:43.711489 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:43.711630 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:43.712634 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "516f5f72767449778f3315b40afc7623"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "469c57b89906f462783688fd6938b586"
server_key_iv: "03380a567b597aef1a0fe9c2e2004668"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.712980 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "516f5f72767449778f3315b40afc7623"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "469c57b89906f462783688fd6938b586"
server_key_iv: "03380a567b597aef1a0fe9c2e2004668"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.714546 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:31:43.715302 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.715487 6036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:43.715577 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "516f5f72767449778f3315b40afc7623"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "469c57b89906f462783688fd6938b586"
server_key_iv: "03380a567b597aef1a0fe9c2e2004668"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.715648 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:43.740152 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:43.742049 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:43.742259 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:43.742511 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:43.742821 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:43.742877 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.742918 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:43.742945 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.749095 6036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44205
I20260620 14:31:43.749135 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44205 every 8 connection(s)
I20260620 14:31:43.749491 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:43.758193 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:43.750201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46227 (local address 127.0.98.62:45825)
0620 14:31:43.750244 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:43.750246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:43.750461 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:31:43.751377 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:43.751378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:43.751387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:43.751445 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:43.751998 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.752497 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.753508 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.753715 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.753764 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:43.755805 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:43.755806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:43.755807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:43.755813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:43.756999 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.757358 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.757359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.757359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.757409 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.757673 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.757674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.757674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.757777 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:43.757834 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:43.758061 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:31:43.758127 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:31:43.758394 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:43.750162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45825 (local address 127.0.98.1:46227)
0620 14:31:43.750336 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:43.750377 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:43.751291 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:43.751501 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:43.751505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:43.751748 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:43.751931 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:43.751934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.752572 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.752573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:43.753370 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:43.753371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.753786 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.753787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:43.753893 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:43.754357 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:43.754365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:43.755710 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:43.757070 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:43.757075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:43.757078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:43.757291 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:43.757518 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:43.757519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:43.757519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:43.757618 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:43.757838 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:43.757843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:43.757955 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:31:43.758036 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:31:43.758151 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260620 14:31:43.758874 6165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45825
I20260620 14:31:43.758945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6036
I20260620 14:31:43.758980 6165 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:43.759042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:43.759183 6165 heartbeater.cc:511] Master 127.0.98.62:45825 requested a full tablet report, sending...
I20260620 14:31:43.759266 392 external_mini_cluster.cc:1428] Setting key 6cb67d92b32cde48521ca2d743129fac
I20260620 14:31:43.759701 5974 ts_manager.cc:194] Registered new tserver with Master: 516f5f72767449778f3315b40afc7623 (127.0.98.1:44205)
I20260620 14:31:43.760401 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46227
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:43.782042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:43.860931 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:43.861138 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:43.861161 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:43.862397 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:43.862447 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:43.862474 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:43.863983 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:43.864249 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:43.864497 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:43.869560 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:43.869589 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:43.869589 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:43.869740 6170 server_base.cc:1034] running on GCE node
I20260620 14:31:43.870047 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:43.870293 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:43.871436 6170 hybrid_clock.cc:648] HybridClock initialized: now 1781965903871425 us; error 29 us; skew 500 ppm
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:43.873965 6170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:43.874459 6170 webserver.cc:460] Webserver started at http://127.0.98.2:45957/ using document root <none> and password file <none>
I20260620 14:31:43.874684 6170 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:43.874754 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:43.874922 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:43.875988 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "688664b469e241948d519df27ca5bd18"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "c446d24d43049d833656ec3f05180906"
server_key_iv: "90176caf28bb5eab4d33a20474dfc2c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.876355 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "688664b469e241948d519df27ca5bd18"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "c446d24d43049d833656ec3f05180906"
server_key_iv: "90176caf28bb5eab4d33a20474dfc2c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.877746 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:43.878499 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.878661 6170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:43.878762 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "688664b469e241948d519df27ca5bd18"
format_stamp: "Formatted at 2026-06-20 14:31:43 on dist-test-slave-tc2g"
server_key: "c446d24d43049d833656ec3f05180906"
server_key_iv: "90176caf28bb5eab4d33a20474dfc2c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:43.878840 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:43.886294 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:43.888056 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:43.888200 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:43.888408 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:43.888713 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:43.888767 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.888813 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:43.888842 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:43.894502 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37663
I20260620 14:31:43.894549 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37663 every 8 connection(s)
I20260620 14:31:43.894846 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:43.896533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6170
I20260620 14:31:43.896618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:43.896847 392 external_mini_cluster.cc:1428] Setting key ee6cf867692eb7a91c7cc6152f32232c
Jun 20 14:31:43 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:31:43.905416 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:43.895526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34457 (local address 127.0.98.62:45825)
0620 14:31:43.895586 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:43.895588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:43.895766 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:31:43.896759 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:43.896760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:43.896774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:43.896839 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:43.897574 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.898141 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.899471 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.899785 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.899883 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:43.902378 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:43.902380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:43.902380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:43.902386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:43.903909 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.904443 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.904445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.904445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.904503 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:43.904778 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:43.904778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:43.904778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:43.904901 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:43.904970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:43.905240 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:31:43.905317 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:43.905622 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:43.895488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45825 (local address 127.0.98.2:34457)
0620 14:31:43.895642 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:43.895680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:43.896573 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:43.896917 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:43.896939 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:43.897268 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:43.897496 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:43.897503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.898240 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.898242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:43.899385 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:43.899387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:43.899890 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:43.899890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:43.899988 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:43.900489 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:43.900505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:43.902285 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:43.903992 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:43.903998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:43.904001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:43.904316 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:43.904584 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:43.904584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:43.904585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:43.904711 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:43.904994 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:43.904999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:43.905112 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:31:43.905221 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:31:43.905388 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260620 14:31:43.906173 6299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45825
I20260620 14:31:43.906279 6299 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:43.906493 6299 heartbeater.cc:511] Master 127.0.98.62:45825 requested a full tablet report, sending...
I20260620 14:31:43.906870 5974 ts_manager.cc:194] Registered new tserver with Master: 688664b469e241948d519df27ca5bd18 (127.0.98.2:37663)
I20260620 14:31:43.907243 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:43.920289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:43.999322 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:43.999562 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:43.999607 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:44.000810 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:44.000878 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:44.000924 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:44.002460 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:44.002728 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:44.002961 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:44.008172 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:44.008186 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:44.008414 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:44.008477 6304 server_base.cc:1034] running on GCE node
I20260620 14:31:44.008693 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:44.008934 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:44.010098 6304 hybrid_clock.cc:648] HybridClock initialized: now 1781965904010081 us; error 50 us; skew 500 ppm
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.012620 6304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:44.013041 6304 webserver.cc:460] Webserver started at http://127.0.98.3:43351/ using document root <none> and password file <none>
I20260620 14:31:44.013275 6304 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:44.013342 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:44.013516 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:44.014555 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c2277a189284241bd0165026c20ff27"
format_stamp: "Formatted at 2026-06-20 14:31:44 on dist-test-slave-tc2g"
server_key: "eb56f028daee8e32c044b91959c03338"
server_key_iv: "068cde6ca4e69b67c04e256e8907bc0b"
server_key_version: "encryptionkey@0"
I20260620 14:31:44.014925 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c2277a189284241bd0165026c20ff27"
format_stamp: "Formatted at 2026-06-20 14:31:44 on dist-test-slave-tc2g"
server_key: "eb56f028daee8e32c044b91959c03338"
server_key_iv: "068cde6ca4e69b67c04e256e8907bc0b"
server_key_version: "encryptionkey@0"
I20260620 14:31:44.016182 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:44.016933 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:44.017149 6304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:44.017251 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "4c2277a189284241bd0165026c20ff27"
format_stamp: "Formatted at 2026-06-20 14:31:44 on dist-test-slave-tc2g"
server_key: "eb56f028daee8e32c044b91959c03338"
server_key_iv: "068cde6ca4e69b67c04e256e8907bc0b"
server_key_version: "encryptionkey@0"
I20260620 14:31:44.017313 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:44.026154 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:44.028102 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:44.028282 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:44.028501 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:44.028853 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:44.028910 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:44.028951 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:44.028980 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:44.034905 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38245
I20260620 14:31:44.034955 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38245 every 8 connection(s)
I20260620 14:31:44.035259 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:44.044190 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:44.035956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41415 (local address 127.0.98.62:45825)
0620 14:31:44.036015 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.036034 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.036216 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:31:44.037182 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.037183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.037196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.037271 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.037891 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.038397 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.039412 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.039609 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.039679 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:44.041829 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:44.041831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:44.041831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:44.041837 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:44.043121 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.043469 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.043469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.043470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.043517 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.043733 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.043733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.043734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.043830 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:44.043880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:44.044077 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:31:44.044128 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:44.044430 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:44.035927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45825 (local address 127.0.98.3:41415)
0620 14:31:44.036090 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:44.036132 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:44.037057 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:44.037328 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:44.037331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:44.037608 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:44.037793 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:44.037798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.038493 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.038495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:44.039310 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:44.039312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.039697 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.039698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:44.039824 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:44.040339 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:44.040347 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:44.041748 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:44.043191 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:44.043196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:44.043198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:44.043404 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:44.043573 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:44.043574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:44.043574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:44.043663 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:44.043884 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:44.043887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:44.043990 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:31:44.044067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:31:44.044174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260620 14:31:44.044749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6304
I20260620 14:31:44.044872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:44.045034 6433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45825
I20260620 14:31:44.045089 392 external_mini_cluster.cc:1428] Setting key c17cda02f0c4a418ea6e933373ea1912
I20260620 14:31:44.045140 6433 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:44.045336 6433 heartbeater.cc:511] Master 127.0.98.62:45825 requested a full tablet report, sending...
I20260620 14:31:44.045747 5973 ts_manager.cc:194] Registered new tserver with Master: 4c2277a189284241bd0165026c20ff27 (127.0.98.3:38245)
I20260620 14:31:44.046201 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41415
I20260620 14:31:44.056897 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.071805 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:31:44.073549 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:44.072715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.0.98.1:44205)
0620 14:31:44.072882 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.072888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.072906 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:31:44.072933 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.072934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.072948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.073023 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.073400 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38552: BlockingRecv error: recv got EOF from 127.0.0.1:38552 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
W20260620 14:31:44.073640 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.074622 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.075531 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:44.083678 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:44.076063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.0.98.1:44205)
0620 14:31:44.076108 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.076109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.076136 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:31:44.076207 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.076207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.076217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.076267 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.076623 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.077347 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.078142 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.078488 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.078546 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:44.080659 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:44.080667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:44.080667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:44.080674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:44.082598 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.082984 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.082987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.082987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.083041 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.083189 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.083189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.083189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.083307 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:44.083403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:44.083470 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:31:44.083529 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:31:44.085487 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:44.084972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.0.98.1:44205)
0620 14:31:44.085006 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.085008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.085040 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:31:44.085097 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.085097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.085106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.085152 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.085396 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38576: BlockingRecv error: recv got EOF from 127.0.0.1:38576 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:31:44.085572 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.086472 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.087263 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:44.093730 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:44.087715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.0.98.1:44205)
0620 14:31:44.087762 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.087769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.087798 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:31:44.087846 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.087846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.087853 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.087909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.088235 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.088722 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.089482 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.089701 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.089754 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:44.091655 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:44.091656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:44.091656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:44.091662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:44.092775 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.093063 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.093064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.093064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.093108 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.093247 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.093247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.093247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.093338 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:44.093402 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:44.093547 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:31:44.093603 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:31:44.095204 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:44.094657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.0.98.1:44205)
0620 14:31:44.094721 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.094723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.094734 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:44.094806 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.094807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.094815 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.094871 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.095145 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38606: BlockingRecv error: recv got EOF from 127.0.0.1:38606 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:31:44.095257 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965876904925-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.096135 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:44.096966 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:31:44.103816 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:44.097441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.98.1:44205)
0620 14:31:44.097478 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:44.097480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:44.097567 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:31:44.097622 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:44.097623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:44.097632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:44.097674 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:44.098070 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.098530 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.099265 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:44.099468 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:44.099551 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:44.101561 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:44.101562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:44.101562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:44.101569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:44.102739 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.103080 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.103081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.103082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.103136 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:44.103332 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:44.103333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:44.103333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:44.103436 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:44.103534 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:44.103639 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:31:44.103710 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:31:44.104585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6036
I20260620 14:31:44.109160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6170
I20260620 14:31:44.113631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6304
I20260620 14:31:44.117808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 5942
2026-06-20T14:31:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1839 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[6453](info): set up 2 sockets
Jun 20 14:31:44 dist-test-slave-tc2g krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260620 14:31:45.165835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:31:45.281656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:45.360517 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:45.360769 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:45.360805 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:45.361995 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:45.362067 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:45.362092 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:45.362121 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:45.362154 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:45.363597 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:45.363885 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:45.364112 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:45.369035 6466 server_base.cc:1034] running on GCE node
W20260620 14:31:45.369004 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.369004 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.368992 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:45.369555 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:45.369863 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:45.371028 6466 hybrid_clock.cc:648] HybridClock initialized: now 1781965905371011 us; error 31 us; skew 500 ppm
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:45.376098 6466 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:31:45.376551 6466 webserver.cc:460] Webserver started at http://127.0.98.62:33171/ using document root <none> and password file <none>
I20260620 14:31:45.376777 6466 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:45.376845 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:45.376996 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:45.378116 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "747e8b96132845e0b6277cd90671ed49"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "dbdf6f84f2834cfc9ba37e840d20b134"
server_key_iv: "8049c7ff61a886587decbe2cabceca41"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.378502 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "747e8b96132845e0b6277cd90671ed49"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "dbdf6f84f2834cfc9ba37e840d20b134"
server_key_iv: "8049c7ff61a886587decbe2cabceca41"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.379846 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:45.380539 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.380821 6466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:45.380936 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "747e8b96132845e0b6277cd90671ed49"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "dbdf6f84f2834cfc9ba37e840d20b134"
server_key_iv: "8049c7ff61a886587decbe2cabceca41"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.381011 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:45.408631 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:45.410567 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:45.410737 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:45.414675 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34027
I20260620 14:31:45.414713 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34027 every 8 connection(s)
I20260620 14:31:45.415076 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:45.416041 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:45.416349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6466
I20260620 14:31:45.416472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:45.416697 392 external_mini_cluster.cc:1428] Setting key f1f545aed8a966d6b18954ae270a9b1e
I20260620 14:31:45.419111 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Bootstrap starting.
I20260620 14:31:45.419795 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:45.420076 6534 log.cc:824] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:45.420758 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: No bootstrap required, opened a new log
I20260620 14:31:45.423070 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } }
I20260620 14:31:45.423283 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:45.423336 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747e8b96132845e0b6277cd90671ed49, State: Initialized, Role: FOLLOWER
I20260620 14:31:45.423467 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } }
I20260620 14:31:45.423552 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:45.423591 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:45.423641 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:45.424286 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } }
I20260620 14:31:45.424399 6534 leader_election.cc:302] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 747e8b96132845e0b6277cd90671ed49; no voters:
I20260620 14:31:45.424558 6534 leader_election.cc:288] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:45.424743 6540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:45.424841 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:45.424909 6540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 LEADER]: Becoming Leader. State: Replica: 747e8b96132845e0b6277cd90671ed49, State: Running, Role: LEADER
I20260620 14:31:45.425001 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } }
I20260620 14:31:45.425349 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } }
I20260620 14:31:45.425529 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:45.425786 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:45.426330 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:45.427088 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747e8b96132845e0b6277cd90671ed49. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } }
I20260620 14:31:45.427201 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:45.428113 6543 catalog_manager.cc:1347] Generated new cluster ID: a5158a4d5fd440e3bd7ee967a4f7c865
I20260620 14:31:45.428160 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:45.428422 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.417227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.0.98.62:34027)
0620 14:31:45.417467 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:45.417477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:45.417514 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:31:45.418043 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:45.418051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:45.418120 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:45.418272 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:45.418853 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.420182 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.421036 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.421691 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.421804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.424237 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:45.424241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:45.424241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:45.424254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:45.426388 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.427082 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.427084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.427085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.427238 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.427517 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.427518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.427518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.427809 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:45.427969 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:45.428103 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:31:45.428239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260620 14:31:45.434938 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:45.435444 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:45.440275 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Generated new TSK 0
I20260620 14:31:45.440474 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:31:45.478467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:45.560762 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:45.561009 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:45.561051 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:45.562299 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:45.562363 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:45.562409 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:45.563876 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:45.564144 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:45.564381 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:45.569278 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.569345 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.569299 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:45.569536 6560 server_base.cc:1034] running on GCE node
I20260620 14:31:45.569720 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:45.569912 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:45.571050 6560 hybrid_clock.cc:648] HybridClock initialized: now 1781965905571037 us; error 28 us; skew 500 ppm
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:45.573513 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:31:45.573961 6560 webserver.cc:460] Webserver started at http://127.0.98.1:43547/ using document root <none> and password file <none>
I20260620 14:31:45.574195 6560 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:45.574265 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:45.574419 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:45.575443 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c518fa4225d4e768fe62c4ecb6ddd88"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "1dcf694eed30537998fc21d0ee3ba508"
server_key_iv: "59a35328578a555291d0f5cdf7568e4e"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.575798 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c518fa4225d4e768fe62c4ecb6ddd88"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "1dcf694eed30537998fc21d0ee3ba508"
server_key_iv: "59a35328578a555291d0f5cdf7568e4e"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.577131 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:45.577893 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.578059 6560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.578163 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "6c518fa4225d4e768fe62c4ecb6ddd88"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "1dcf694eed30537998fc21d0ee3ba508"
server_key_iv: "59a35328578a555291d0f5cdf7568e4e"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.578236 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:45.590979 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:45.592931 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:45.593096 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:45.593319 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:45.593703 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:45.593762 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.593811 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:45.593838 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.599402 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43379
I20260620 14:31:45.599476 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43379 every 8 connection(s)
I20260620 14:31:45.599789 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:45.604146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6560
I20260620 14:31:45.604236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:45.604452 392 external_mini_cluster.cc:1428] Setting key 37e54364c71a7953b2d60bfac4118f22
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:45.609534 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.600499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58843 (local address 127.0.98.62:34027)
0620 14:31:45.600541 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:45.600543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:45.600754 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:31:45.601199 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:45.601200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:45.601210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:45.601257 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:45.602180 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.602676 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.603644 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.603844 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.603927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.606482 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:45.606483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:45.606484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:45.606492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:45.607936 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.608376 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.608377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.608378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.608439 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.608707 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.608708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.608708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.608827 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:45.608915 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:45.609275 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:31:45.609369 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:45.609795 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.600482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:58843)
0620 14:31:45.600632 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:45.600666 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:45.601119 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:45.601316 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:45.601319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:45.601913 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:45.602100 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.602105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.602757 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.602757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.603544 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.603546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.603960 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.603962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.604088 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.604664 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:45.604673 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:45.606385 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:45.608046 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.608053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.608056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.608285 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.608507 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.608508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.608508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.608627 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.608926 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:45.608932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:45.609116 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:31:45.609246 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:31:45.609497 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260620 14:31:45.610453 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:45.610541 6689 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:45.610750 6689 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:45.611289 6498 ts_manager.cc:194] Registered new tserver with Master: 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379)
I20260620 14:31:45.611901 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58843
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:31:45.628403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:45.709088 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:45.709295 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:45.709319 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:45.710472 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:45.710517 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:45.710544 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:45.711910 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:45.712090 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:45.712260 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:45.717080 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.717080 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.717118 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:45.717365 6694 server_base.cc:1034] running on GCE node
I20260620 14:31:45.717597 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:45.717792 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:45.718925 6694 hybrid_clock.cc:648] HybridClock initialized: now 1781965905718899 us; error 41 us; skew 500 ppm
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:45.721519 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:31:45.721932 6694 webserver.cc:460] Webserver started at http://127.0.98.2:37287/ using document root <none> and password file <none>
I20260620 14:31:45.722155 6694 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:45.722224 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:45.722386 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:45.723412 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "12fa50ee3f2e4d539ee4ca45a05f952a"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "651f80e43d9f5736f7d3c99f51f9c9d5"
server_key_iv: "089def6dd3d50423683cac5268ba3b1f"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.723776 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12fa50ee3f2e4d539ee4ca45a05f952a"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "651f80e43d9f5736f7d3c99f51f9c9d5"
server_key_iv: "089def6dd3d50423683cac5268ba3b1f"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.725083 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:45.725941 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.726236 6694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:45.726403 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "12fa50ee3f2e4d539ee4ca45a05f952a"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "651f80e43d9f5736f7d3c99f51f9c9d5"
server_key_iv: "089def6dd3d50423683cac5268ba3b1f"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.726475 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:45.738718 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:45.740473 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:45.740639 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:45.740885 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:45.741190 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:45.741247 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.741290 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:45.741319 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.746975 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40885
I20260620 14:31:45.747042 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40885 every 8 connection(s)
I20260620 14:31:45.747422 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:45.753153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6694
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:45.753285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:45.753554 392 external_mini_cluster.cc:1428] Setting key 4f35aace17b57d1cddf9e3b57bd3e3ff
I20260620 14:31:45.756755 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.748162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42917 (local address 127.0.98.62:34027)
0620 14:31:45.748232 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:45.748235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:45.748445 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:31:45.748915 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:45.748916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:45.748926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:45.748979 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:45.750000 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.750498 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.751460 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.751647 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.751704 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.753882 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:45.753884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:45.753885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:45.753893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:45.755271 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.755773 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.755776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.755777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.755838 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.756101 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.756102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.756102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.756221 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:45.756307 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:45.756579 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:31:45.756661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:45.756986 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.748145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:42917)
0620 14:31:45.748318 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:45.748353 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:45.748831 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:45.749033 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:45.749037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:45.749715 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:45.749913 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.749918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.750577 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.750578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.751375 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.751376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.751721 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.751721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.751819 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.752284 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:45.752292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:45.753763 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:45.755353 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.755360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.755362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.755684 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.755909 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.755910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.755910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.756030 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.756292 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:45.756298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:45.756447 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:31:45.756558 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:31:45.756722 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260620 14:31:45.757602 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:45.757696 6823 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:45.757890 6823 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:45.758294 6498 ts_manager.cc:194] Registered new tserver with Master: 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
I20260620 14:31:45.758879 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42917
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:31:45.778218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:45.858891 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:45.859108 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:45.859133 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:45.860222 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:45.860257 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:45.860281 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:45.861750 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:45.861935 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:45.862110 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:45.866954 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.866983 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:45.866983 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:45.867702 6828 server_base.cc:1034] running on GCE node
I20260620 14:31:45.867923 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:45.868175 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:45.869316 6828 hybrid_clock.cc:648] HybridClock initialized: now 1781965905869306 us; error 28 us; skew 500 ppm
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:45.871872 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:31:45.872331 6828 webserver.cc:460] Webserver started at http://127.0.98.3:34893/ using document root <none> and password file <none>
I20260620 14:31:45.872587 6828 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:45.872633 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:45.872787 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:45.873924 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "56eaa369abcbf80e8e884e2b38413d96"
server_key_iv: "90c4d14dc1a79ea4737d4e24343d97c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.874293 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "56eaa369abcbf80e8e884e2b38413d96"
server_key_iv: "90c4d14dc1a79ea4737d4e24343d97c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.875630 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:45.876367 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.876544 6828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:45.876652 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "56eaa369abcbf80e8e884e2b38413d96"
server_key_iv: "90c4d14dc1a79ea4737d4e24343d97c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:45.876729 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:45.885605 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:45.887600 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:45.887769 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:45.888013 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:45.888334 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:45.888389 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.888432 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:45.888456 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:45.894338 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42343
I20260620 14:31:45.894384 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42343 every 8 connection(s)
I20260620 14:31:45.894694 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:31:45 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:45.903054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6828
I20260620 14:31:45.903168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:45.903373 392 external_mini_cluster.cc:1428] Setting key 7cc0894381e1d224a4a26401126b17bc
I20260620 14:31:45.904122 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.895402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51627 (local address 127.0.98.62:34027)
0620 14:31:45.895480 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:45.895482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:45.895659 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:31:45.896180 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:45.896181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:45.896192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:45.896260 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:45.897219 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.897826 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.898837 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.899049 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.899121 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.901357 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:45.901359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:45.901359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:45.901366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:45.902629 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.903061 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.903063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.903063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.903127 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:45.903407 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:45.903408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:45.903408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:45.903570 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:45.903651 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:45.903956 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:31:45.904036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:31:45.904306 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:45.895363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:51627)
0620 14:31:45.895538 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:45.895580 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:45.896085 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:45.896326 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:45.896330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:45.896932 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:45.897123 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.897127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.897917 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.897918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.898727 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:45.898729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:45.899132 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:45.899133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:45.899226 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:45.899844 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:45.899853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:45.901243 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:45.902700 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.902705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.902707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.902955 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.903196 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:45.903197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:45.903198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:45.903330 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:45.903640 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:45.903646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:45.903802 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:31:45.903922 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:31:45.904077 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260620 14:31:45.904687 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:45.904820 6957 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:45.904999 6957 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:45.905370 6498 ts_manager.cc:194] Registered new tserver with Master: fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:45.905776 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51627
I20260620 14:31:45.915007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:31:45.938374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43929
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34027,127.0.98.61:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:46.019466 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:46.019675 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:46.019699 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:46.020860 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:46.020902 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:46.020917 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:46.020931 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:46.020942 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:46.022483 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34027,127.0.98.61:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
W20260620 14:31:46.022589 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34027,127.0.98.61:43929'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:31:46.022773 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:46.023025 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.027727 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.027742 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.028029 6962 server_base.cc:1034] running on GCE node
W20260620 14:31:46.027731 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.028306 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.028599 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.029754 6962 hybrid_clock.cc:648] HybridClock initialized: now 1781965906029744 us; error 50 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.032155 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:31:46.032634 6962 webserver.cc:460] Webserver started at http://127.0.98.61:44293/ using document root <none> and password file <none>
I20260620 14:31:46.032857 6962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.032925 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.033077 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:46.034202 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/instance:
uuid: "16e53c4e15f9461dae98622154f1c21a"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "e4e9f373e89ed4d653bf1bf3962f7791"
server_key_iv: "bfba88a19571426a659d6207984b04cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.034565 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal/instance:
uuid: "16e53c4e15f9461dae98622154f1c21a"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "e4e9f373e89ed4d653bf1bf3962f7791"
server_key_iv: "bfba88a19571426a659d6207984b04cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.035882 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:46.036617 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.036798 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:46.036901 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "16e53c4e15f9461dae98622154f1c21a"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "e4e9f373e89ed4d653bf1bf3962f7791"
server_key_iv: "bfba88a19571426a659d6207984b04cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.036972 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.054908 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.057127 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:46.057320 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.061394 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43929
I20260620 14:31:46.061496 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43929 every 8 connection(s)
I20260620 14:31:46.061885 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:31:46.062870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 6962
I20260620 14:31:46.062973 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:46.065311 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:46.073587 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.063511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.0.98.61:43929)
0620 14:31:46.063692 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.063701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.063742 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:31:46.064163 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.064172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.064240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.064396 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.064953 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.066208 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.067035 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.067443 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.067558 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.069910 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.069915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.069916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.069921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.071848 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.072178 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.072181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.072182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.072282 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.072593 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.072594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.072594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.072878 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.073016 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.073165 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.073290 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260620 14:31:46.073763 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.066031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.62:34027)
0620 14:31:46.066096 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.066098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.066279 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:31:46.066378 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.066379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.066390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.066458 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.066907 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.067385 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.068358 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.068548 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.068628 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.070774 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.070775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.070775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.070781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.072257 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.072779 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.072780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.072780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.072857 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.073108 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.073109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.073109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.073291 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.073364 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.073596 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.073678 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:46.073794 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.065969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39008)
0620 14:31:46.066162 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.066197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.066296 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.066475 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.066478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.066751 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.066815 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.066816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.067480 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.067481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.068251 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.068254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.068638 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.068639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.068726 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.069167 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.069175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.070696 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.072390 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.072397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.072399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.072713 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.072963 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.072964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.072964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.073032 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.073374 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.073380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.073466 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.073549 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.073631 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260620 14:31:46.074443 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:31:46.080188 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.074747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42666)
0620 14:31:46.074774 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.074783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.074819 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.074925 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.074925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.075031 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.075072 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.075074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.075614 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.075614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.076073 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.076074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.076383 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.076383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.076431 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.076778 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.076782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.077925 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.079315 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.079316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.079316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.079486 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.079636 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.079637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.079637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.079693 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.079948 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.079948 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.079981 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.080036 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.080107 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 14:31:46.080236 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.074773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.61:43929)
0620 14:31:46.074825 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.074827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.074836 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:46.074854 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.074855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.074863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.074922 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.075113 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.075557 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.076137 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.076326 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.076375 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.077998 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.077999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.077999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.078005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.079237 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.079540 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.079541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.079541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.079584 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.079742 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.079743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.079743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.079885 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.079941 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.080051 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.080149 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":35}
I20260620 14:31:46.081971 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Bootstrap starting.
I20260620 14:31:46.082760 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:46.083032 7030 log.cc:824] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:46.083745 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: No bootstrap required, opened a new log
I20260620 14:31:46.085088 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } }
I20260620 14:31:46.085244 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:46.085294 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e53c4e15f9461dae98622154f1c21a, State: Initialized, Role: FOLLOWER
I20260620 14:31:46.085413 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } }
I20260620 14:31:46.085743 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:46.085708 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } }
I20260620 14:31:46.085836 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:31:46.087464 7047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:31:46.087527 7047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:31:46.089051 6962 master_runner.cc:428] Detected that this master 16e53c4e15f9461dae98622154f1c21a is joining an existing cluster
I20260620 14:31:46.089102 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43929
I20260620 14:31:46.096446 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.090150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42668)
0620 14:31:46.090507 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.090514 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.090571 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.090700 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.090700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.090899 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.090969 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.090971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.091618 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.091619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.092183 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.092184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.092552 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.092553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.092619 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.093082 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.093095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.093387 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.095330 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.095331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.095332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.095518 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.095735 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.095736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.095736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.095792 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.096028 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.096029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.096079 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.096159 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.096316 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260620 14:31:46.096534 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.090238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.98.61:43929)
0620 14:31:46.090331 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.090333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.090567 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:31:46.090626 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.090627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.090636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.090689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.091045 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.091540 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.092275 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.092463 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.092540 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.093748 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.093749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.093750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.093756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.095243 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.095609 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.095610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.095610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.095661 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.095851 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.095852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.095853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.095964 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.096036 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.096180 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.096397 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":75}
I20260620 14:31:46.097527 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.090288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.0.98.62:34027)
0620 14:31:46.090356 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.090358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.090446 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:31:46.090507 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.090508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.090520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.090573 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.091111 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.091877 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.092910 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.093200 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.093252 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.094236 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.094249 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.094250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.094257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.096021 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.096482 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.096483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.096483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.096541 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.096949 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.096950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.096951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.097072 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.097114 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.097328 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.097389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:46.097543 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.090141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39020)
0620 14:31:46.090381 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.090389 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.090450 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.090622 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.090623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.090877 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.090997 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.091000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.091978 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.091979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.092810 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.092812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.093328 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.093329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.093390 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.093760 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.093765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.094133 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.096118 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.096119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.096119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.096395 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.096815 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.096815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.096816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.096875 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.097160 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.097161 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.097212 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.097291 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.097382 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:31:46.107334 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.100030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42674)
0620 14:31:46.100196 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.100241 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.100302 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.100497 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.100502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.100738 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.100782 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.100784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.101653 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.101654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.102221 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.102222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.102676 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.102677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.102729 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.103190 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.103195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.104710 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.106352 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.106359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.106361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.106564 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.106763 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.106764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.106764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.106816 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.107058 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.107064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.107106 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.107171 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.107262 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260620 14:31:46.107376 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.100086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.98.61:43929)
0620 14:31:46.100126 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.100128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.100306 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:31:46.100412 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.100413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.100423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.100491 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.100852 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.101583 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.102298 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.102603 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.102672 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.104796 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.104797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.104797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.104804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.106278 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.106635 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.106636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.106637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.106692 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.106872 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.106873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.106873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.106998 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.107063 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.107211 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.107286 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:46.107946 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43929: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "16e53c4e15f9461dae98622154f1c21a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43929
}
attrs {
promote: true
}
}
dest_uuid: "747e8b96132845e0b6277cd90671ed49"
cas_config_opid_index: -1
I20260620 14:31:46.108273 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } }
I20260620 14:31:46.108781 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16e53c4e15f9461dae98622154f1c21a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.109074 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:46.109840 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43929
I20260620 14:31:46.110054 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:31:46.110060 6962 master.cc:537] Master@127.0.98.61:43929 shutting down...
I20260620 14:31:46.110603 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.110697 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:46.110787 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:31:46.111451 6484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:31:46.112118 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 -> Peer 16e53c4e15f9461dae98622154f1c21a (127.0.98.61:43929): Couldn't send request to peer 16e53c4e15f9461dae98622154f1c21a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:31:46.113010 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:46.113096 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:46.113166 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: stopping tablet replica
I20260620 14:31:46.125900 6962 master.cc:552] Master@127.0.98.61:43929 shutdown complete.
I20260620 14:31:46.125952 6962 master_runner.cc:305] Clearing existing system tablet
I20260620 14:31:46.126093 6962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.126147 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.126955 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:46.127467 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.127595 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.127658 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.127698 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "16e53c4e15f9461dae98622154f1c21a"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "e4e9f373e89ed4d653bf1bf3962f7791"
server_key_iv: "bfba88a19571426a659d6207984b04cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.128350 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:46.129696 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:31:46.129753 6962 log.cc:1196] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:31:46.129962 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Deleting consensus metadata
I20260620 14:31:46.130089 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:34027
I20260620 14:31:46.130688 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34027
I20260620 14:31:46.137337 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.131002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39034)
0620 14:31:46.131119 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.131130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.131193 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.131332 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.131333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.131489 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.131544 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.131546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.132506 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.132507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.133089 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.133091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.133693 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.133694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.133751 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.134307 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.134312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.134617 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.136318 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.136319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.136319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.136511 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.136713 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.136713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.136714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.136764 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.137016 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.137016 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.137056 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.137120 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.137253 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":69,"threads_started":1}
I20260620 14:31:46.137359 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.131070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.98.62:34027)
0620 14:31:46.131120 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.131123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.131185 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:31:46.131239 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.131240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.131250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.131329 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.131619 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.132398 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.133177 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.133570 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.133645 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.134707 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.134708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.134709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.134724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.136231 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.136587 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.136588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.136589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.136647 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.136824 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.136825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.136825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.136956 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.137023 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.137189 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.137266 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:31:46.138000 6518 tablet_copy_service.cc:140] P 747e8b96132845e0b6277cd90671ed49: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 16e53c4e15f9461dae98622154f1c21a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39034)
I20260620 14:31:46.138110 6518 tablet_copy_service.cc:161] P 747e8b96132845e0b6277cd90671ed49: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 16e53c4e15f9461dae98622154f1c21a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39034: session id = 16e53c4e15f9461dae98622154f1c21a-00000000000000000000000000000000
I20260620 14:31:46.138743 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:31:46.139371 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:31:46.140318 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: tablet copy: Starting download of 0 data blocks...
I20260620 14:31:46.140481 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: tablet copy: Starting download of 1 WAL segments...
I20260620 14:31:46.141517 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:31:46.142378 6518 tablet_copy_service.cc:342] P 747e8b96132845e0b6277cd90671ed49: Request end of tablet copy session 16e53c4e15f9461dae98622154f1c21a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39034
I20260620 14:31:46.142448 6518 tablet_copy_service.cc:433] P 747e8b96132845e0b6277cd90671ed49: ending tablet copy session 16e53c4e15f9461dae98622154f1c21a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 16e53c4e15f9461dae98622154f1c21a
I20260620 14:31:46.144898 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.146042 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.146159 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.146174 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.146303 6962 server_base.cc:1034] running on GCE node
I20260620 14:31:46.146436 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.146564 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.147689 6962 hybrid_clock.cc:648] HybridClock initialized: now 1781965906147690 us; error 32 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.149084 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:31:46.149335 6962 webserver.cc:460] Webserver started at http://127.0.98.61:36847/ using document root <none> and password file <none>
I20260620 14:31:46.149472 6962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.149511 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.150190 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:46.150718 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.150839 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:46.150888 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "16e53c4e15f9461dae98622154f1c21a"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "e4e9f373e89ed4d653bf1bf3962f7791"
server_key_iv: "bfba88a19571426a659d6207984b04cc"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.150943 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.160010 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.161671 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.164804 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43929
I20260620 14:31:46.164999 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43929 every 8 connection(s)
I20260620 14:31:46.165210 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:31:46.166141 7131 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:31:46.166626 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Bootstrap starting.
I20260620 14:31:46.169497 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:31:46.169746 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: Bootstrap complete.
I20260620 14:31:46.169978 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } }
I20260620 14:31:46.170063 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16e53c4e15f9461dae98622154f1c21a, State: Initialized, Role: LEARNER
I20260620 14:31:46.170099 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } }
I20260620 14:31:46.170269 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:46.170289 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.170410 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:31:46.171785 7145 catalog_manager.cc:1259] Loaded cluster ID: a5158a4d5fd440e3bd7ee967a4f7c865
I20260620 14:31:46.171902 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: loading cluster ID for follower catalog manager: success
I20260620 14:31:46.172870 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: acquiring CA information for follower catalog manager: success
I20260620 14:31:46.173166 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:31:46.182641 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.175508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.98.61:43929)
0620 14:31:46.175645 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.175647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.175655 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:46.175679 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.175680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.175691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.175768 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.176331 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.176977 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.177816 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.178033 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.178112 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.179533 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.179535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.179535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.179543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.181354 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.181744 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.181745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.181745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.181829 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.182054 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.182055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.182055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.182181 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.182289 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.182423 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.182516 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 14:31:46.209810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46209
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:46.298609 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:46.298866 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:46.298906 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:46.300107 7149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:46.300179 7149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:46.300206 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:46.300238 7149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:46.300262 7149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:46.301870 7149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:46.302140 7149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:46.302377 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.307345 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.307335 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.307457 7149 server_base.cc:1034] running on GCE node
W20260620 14:31:46.307339 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.307834 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.308133 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.309283 7149 hybrid_clock.cc:648] HybridClock initialized: now 1781965906309266 us; error 36 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.311957 7149 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:31:46.312425 7149 webserver.cc:460] Webserver started at http://127.0.98.60:41391/ using document root <none> and password file <none>
I20260620 14:31:46.312667 7149 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.312714 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.312873 7149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:46.314023 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/instance:
uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "f74a54bf579724415839cf5bc7568295"
server_key_iv: "19c132760f85392e153bb6ebdcf67bc5"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.314393 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal/instance:
uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "f74a54bf579724415839cf5bc7568295"
server_key_iv: "19c132760f85392e153bb6ebdcf67bc5"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.315713 7149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:46.316501 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.316689 7149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:46.316798 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "f74a54bf579724415839cf5bc7568295"
server_key_iv: "19c132760f85392e153bb6ebdcf67bc5"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.316877 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.341645 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.343703 7149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:46.343870 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.347580 7149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46209
I20260620 14:31:46.347637 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46209 every 8 connection(s)
I20260620 14:31:46.348011 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:31:46.349156 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:46.351284 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:31:46.356316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7149
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:46.360020 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.352100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.98.62:34027)
0620 14:31:46.352157 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.352159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.352279 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:31:46.352714 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.352715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.352724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.352788 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.353355 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.353889 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.354829 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.355014 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.355064 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.357263 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.357265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.357265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.357271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.358655 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.359066 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.359068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.359068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.359132 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.359389 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.359390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.359390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.359503 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.359579 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.359848 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.359934 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:31:46.360270 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.351997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39046)
0620 14:31:46.352150 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.352191 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.352630 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.352844 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.352848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.353092 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.353287 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.353291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.353963 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.353964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.354760 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.354762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.355082 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.355083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.355182 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.355624 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.355631 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.357156 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.358755 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.358762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.358764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.358987 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.359202 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.359203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.359203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.359313 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.359570 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.359576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.359718 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.359827 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.359970 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260620 14:31:46.360841 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:31:46.365334 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.356827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.0.98.60:46209)
0620 14:31:46.356962 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.356967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.356984 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:46.357012 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.357013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.357028 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.357093 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.357733 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.358559 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.359315 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.359615 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.359707 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.362626 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.362631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.362631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.362637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.364318 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.364637 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.364640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.364641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.364700 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.364915 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.364916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.364916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.365051 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.365174 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.365215 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.365270 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260620 14:31:46.367847 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.361107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42708)
0620 14:31:46.361159 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.361170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.361211 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.361348 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.361349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.361529 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.361580 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.361581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.362435 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.362436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.363009 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.363011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.363397 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.363398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.363469 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.363804 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.363808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.365691 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.367162 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.367163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.367163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.367314 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.367468 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.367469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.367469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.367507 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.367659 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.367660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.367688 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.367731 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.367791 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:31:46.367869 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.361162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.0.98.61:43929)
0620 14:31:46.361219 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.361221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.361250 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:31:46.361275 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.361276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.361287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.361325 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.361655 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.362340 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.363095 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.363310 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.363389 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.365781 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.365782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.365782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.365790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.367094 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.367378 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.367379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.367379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.367426 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.367540 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.367541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.367541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.367626 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.367668 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.367746 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.367808 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:46.368264 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:31:46.373975 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.368526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.0.1:40682)
0620 14:31:46.368572 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.368583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.368621 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.368740 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.368741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.368843 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.368871 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.368872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.369489 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.369489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.369981 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.369982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.370341 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.370342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.370377 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.370731 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.370735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.371926 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.373150 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.373151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.373151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.373302 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.373516 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.373516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.373517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.373576 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.373770 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.373771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.373801 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.373846 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.373917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:31:46.374119 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.368567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.0.98.60:46209)
0620 14:31:46.368600 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.368604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.368615 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:46.368679 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.368680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.368687 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.368737 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.368933 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.369374 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.370039 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.370276 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.370342 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.371993 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.371994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.371994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.372000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.373080 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.373365 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.373366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.373366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.373410 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.373629 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.373629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.373630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.373718 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.373766 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.373871 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.373972 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":17}
I20260620 14:31:46.375384 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Bootstrap starting.
I20260620 14:31:46.376168 7217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:46.376480 7217 log.cc:824] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:46.377185 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: No bootstrap required, opened a new log
I20260620 14:31:46.378657 7217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } }
I20260620 14:31:46.378791 7217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:46.378814 7217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6a51e8f72d4b9ea8e18fc46b96026c, State: Initialized, Role: FOLLOWER
I20260620 14:31:46.378944 7217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } }
I20260620 14:31:46.379202 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } } }
I20260620 14:31:46.379266 7217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:46.379274 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:31:46.381534 7233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:31:46.381623 7233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:31:46.383934 7149 master_runner.cc:428] Detected that this master 9f6a51e8f72d4b9ea8e18fc46b96026c is joining an existing cluster
I20260620 14:31:46.384003 7149 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46209
I20260620 14:31:46.406111 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.0.98.60:46209)
0620 14:31:46.385395 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.385397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.385536 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:31:46.385618 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.385619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.385627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.385653 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.386636 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387367 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388411 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388684 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388736 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.402643 (+ 13907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.402645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.402645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.402652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.404580 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405289 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.405344 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405565 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.405706 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.405750 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.405925 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.406006 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:46.406210 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.0.98.61:43929)
0620 14:31:46.385149 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.385151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.385518 (+ 367us) server_negotiation.cc:407] Connection header received
0620 14:31:46.385609 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.385610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.385620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.385650 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.386577 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387355 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388565 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388876 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388915 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.402436 (+ 13521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.402439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.402439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.402448 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.404426 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405279 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.405343 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405613 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.405766 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.405809 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.406026 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.406115 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:46.406445 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.0.1:40688)
0620 14:31:46.385479 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.385489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.385556 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.385705 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.385706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.386478 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.386545 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.386547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387455 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.387455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.388310 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.388312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388814 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.388869 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.389327 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.389332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.402533 (+ 13201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.404667 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.404668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.404668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405178 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.405408 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.405409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405499 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.405792 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.405793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.405831 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.406251 (+ 420us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.406347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260620 14:31:46.406548 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42714)
0620 14:31:46.385463 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.385472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.385540 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.385709 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.385710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.386442 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.386493 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.386495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387454 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.387455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.388451 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.388453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388996 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.389341 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.389809 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.389813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.402266 (+ 12453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.404647 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.404649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.404649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405175 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.405444 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.405445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405525 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.405854 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.405854 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.405911 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.406344 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.406451 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":40,"thread_start_us":85,"threads_started":1}
I20260620 14:31:46.407207 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39050)
0620 14:31:46.385321 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.385333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.385439 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.385644 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.385644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.386374 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.386462 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.386465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387454 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.387455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.388451 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.388453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388992 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.389052 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.402175 (+ 13123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.402183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.402828 (+ 645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.405322 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.405323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405588 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.405811 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.405813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.405866 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.406148 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.406149 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.406961 (+ 812us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.407026 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.407118 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":45,"thread_start_us":117,"threads_started":1}
I20260620 14:31:46.407286 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.385041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.62:34027)
0620 14:31:46.385084 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.385086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.385387 (+ 301us) server_negotiation.cc:407] Connection header received
0620 14:31:46.385527 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.385528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.385541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.385574 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.386564 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.387352 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388564 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.388873 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.388915 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.402966 (+ 14051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.402968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.402969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.402977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.405239 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405677 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.405731 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.405939 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.405940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.405941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.406066 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.406117 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.407101 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.407185 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:31:46.416898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6560
I20260620 14:31:46.417547 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.410303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.0.1:40696)
0620 14:31:46.410365 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.410376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.410420 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.410550 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.410550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.410820 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.410867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.410870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.411549 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.411550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.412117 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.412119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.412470 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.412472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.412549 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.412934 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.412939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.414248 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.416145 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.416146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.416147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.416396 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.416606 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.416607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.416607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.416675 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.417159 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.417160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.417207 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.417270 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.417413 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:31:46.418094 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.410359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.98.60:46209)
0620 14:31:46.410401 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.410403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.410412 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:46.410491 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.410491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.410501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.410548 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.410972 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.411457 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.412223 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.412400 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.412448 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.414349 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.414350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.414350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.414363 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.416051 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.416485 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.416486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.416486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.416542 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.416833 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.416835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.416835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.417052 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.417128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.417928 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.418007 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:46.422780 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46209: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46209
}
attrs {
promote: true
}
}
dest_uuid: "747e8b96132845e0b6277cd90671ed49"
cas_config_opid_index: 5
I20260620 14:31:46.423099 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
I20260620 14:31:46.423494 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.423537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43379
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34027
--webserver_port=43547
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:31:46.423786 7191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:46.424604 7191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:31:46.424628 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46209
I20260620 14:31:46.424705 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.424785 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:46.424957 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:46.426339 7191 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
W20260620 14:31:46.426405 7191 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
I20260620 14:31:46.426899 7191 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 16e53c4e15f9461dae98622154f1c21a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.428078 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747e8b96132845e0b6277cd90671ed49. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.428167 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:31:46.428217 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.428268 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:31:46.428305 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } }
I20260620 14:31:46.428339 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:31:46.429221 7149 master.cc:537] Master@127.0.98.60:46209 shutting down...
I20260620 14:31:46.429360 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.429750 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
W20260620 14:31:46.429903 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 -> Peer 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60:46209): Couldn't send request to peer 9f6a51e8f72d4b9ea8e18fc46b96026c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:31:46.430872 7243 mvcc.cc:205] Tried to move back new op lower bound from 7298932348681519104 to 7298932348621082624. Current Snapshot: MvccSnapshot[applied={T|T < 7298932348630310912}]
I20260620 14:31:46.430930 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.425093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42726)
0620 14:31:46.425216 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.425228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.425268 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.425548 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.425549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.425551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:46.425601 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.425602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.426610 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.426611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.427894 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.427898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.430581 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.430582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.430670 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.430698 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.430754 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.430815 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 14:31:46.432116 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.425179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.0.98.61:43929)
0620 14:31:46.425309 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.425311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.425318 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:46.425342 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.425343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.425367 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.425472 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:46.425672 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.426482 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.428245 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.428984 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.429064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.429151 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.431965 (+ 2814us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.432023 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:31:46.432399 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 LEARNER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:31:46.432621 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:31:46.433238 7149 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 14:31:46.433157 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.433358 7149 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 14:31:46.433398 7149 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: stopping tablet replica
I20260620 14:31:46.434278 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.434437 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:31:46.434490 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747e8b96132845e0b6277cd90671ed49. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: true } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.434580 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:31:46.435117 7057 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: attempting to promote NON_VOTER 16e53c4e15f9461dae98622154f1c21a to VOTER
I20260620 14:31:46.435322 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
W20260620 14:31:46.435963 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 -> Peer 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60:46209): Couldn't send request to peer 9f6a51e8f72d4b9ea8e18fc46b96026c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:31:46.436182 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 LEARNER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:31:46.436383 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:31:46.437701 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16e53c4e15f9461dae98622154f1c21a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.437826 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 16e53c4e15f9461dae98622154f1c21a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.438642 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.438714 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:46.438778 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.438822 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:46.438931 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.439007 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:46.452896 7149 master.cc:552] Master@127.0.98.60:46209 shutdown complete.
I20260620 14:31:46.452968 7149 master_runner.cc:305] Clearing existing system tablet
I20260620 14:31:46.453104 7149 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.453154 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.453995 7149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:31:46.454612 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.454787 7149 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.454849 7149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:46.454873 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "f74a54bf579724415839cf5bc7568295"
server_key_iv: "19c132760f85392e153bb6ebdcf67bc5"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.455561 7149 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:46.457048 7149 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:31:46.457118 7149 log.cc:1196] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:31:46.457347 7149 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Deleting consensus metadata
I20260620 14:31:46.457494 7149 master_runner.cc:315] Copying system tablet from 127.0.98.62:34027
I20260620 14:31:46.458092 7149 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34027
I20260620 14:31:46.465513 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.458533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.98.62:34027)
0620 14:31:46.458580 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.458582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.458733 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:31:46.458796 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.458798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.458811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.458875 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.459221 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.459979 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.460987 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.461283 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.461363 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.462518 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.462520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.462520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.462528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.464218 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.464611 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.464613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.464614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.464677 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.464941 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.464942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.464942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.465074 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.465142 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.465303 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.465388 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:46.465512 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.458448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39064)
0620 14:31:46.458632 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.458644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.458725 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.458879 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.458880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.459083 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.459141 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.459143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.460080 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.460081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.460893 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.460896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.461373 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.461374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.461475 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.462011 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.462017 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.462419 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.464309 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.464310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.464310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.464532 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.464801 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.464801 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.464802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.464869 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.465146 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.465147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.465194 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.465269 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.465378 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260620 14:31:46.466168 6518 tablet_copy_service.cc:140] P 747e8b96132845e0b6277cd90671ed49: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9f6a51e8f72d4b9ea8e18fc46b96026c ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39064)
I20260620 14:31:46.466228 6518 tablet_copy_service.cc:161] P 747e8b96132845e0b6277cd90671ed49: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9f6a51e8f72d4b9ea8e18fc46b96026c at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39064: session id = 9f6a51e8f72d4b9ea8e18fc46b96026c-00000000000000000000000000000000
I20260620 14:31:46.466809 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:31:46.467314 7149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:31:46.468587 7149 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: tablet copy: Starting download of 0 data blocks...
I20260620 14:31:46.468736 7149 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: tablet copy: Starting download of 1 WAL segments...
I20260620 14:31:46.469604 7149 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:31:46.470803 6518 tablet_copy_service.cc:342] P 747e8b96132845e0b6277cd90671ed49: Request end of tablet copy session 9f6a51e8f72d4b9ea8e18fc46b96026c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39064
I20260620 14:31:46.470862 6518 tablet_copy_service.cc:433] P 747e8b96132845e0b6277cd90671ed49: ending tablet copy session 9f6a51e8f72d4b9ea8e18fc46b96026c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9f6a51e8f72d4b9ea8e18fc46b96026c
I20260620 14:31:46.473610 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.475557 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.478471 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.478647 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.485610 7149 server_base.cc:1034] running on GCE node
I20260620 14:31:46.485754 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.485898 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.487025 7149 hybrid_clock.cc:648] HybridClock initialized: now 1781965906487003 us; error 42 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.488554 7149 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:31:46.488862 7149 webserver.cc:460] Webserver started at http://127.0.98.60:34729/ using document root <none> and password file <none>
I20260620 14:31:46.488987 7149 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.489028 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.489846 7149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:46.490427 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.490597 7149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:46.490655 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c"
format_stamp: "Formatted at 2026-06-20 14:31:46 on dist-test-slave-tc2g"
server_key: "f74a54bf579724415839cf5bc7568295"
server_key_iv: "19c132760f85392e153bb6ebdcf67bc5"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.490718 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.509479 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.511399 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.514707 7149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46209
I20260620 14:31:46.514842 7323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46209 every 8 connection(s)
I20260620 14:31:46.515053 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:31:46.515966 7324 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:31:46.516505 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Bootstrap starting.
I20260620 14:31:46.518918 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:31:46.519130 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Bootstrap complete.
I20260620 14:31:46.519325 7324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
I20260620 14:31:46.519418 7324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f6a51e8f72d4b9ea8e18fc46b96026c, State: Initialized, Role: LEARNER
I20260620 14:31:46.519455 7324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } }
I20260620 14:31:46.519645 7324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:46.521117 7338 catalog_manager.cc:1259] Loaded cluster ID: a5158a4d5fd440e3bd7ee967a4f7c865
I20260620 14:31:46.521214 7338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: loading cluster ID for follower catalog manager: success
I20260620 14:31:46.522253 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:46.522351 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: LEARNER
I20260620 14:31:46.522550 7338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: acquiring CA information for follower catalog manager: success
I20260620 14:31:46.522846 7338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:31:46.526361 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:46.526567 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:46.526607 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:46.527822 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:46.527894 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:46.527938 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:46.529557 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43547
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:46.529853 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:46.530097 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.534861 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.534898 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.534992 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.535254 7242 server_base.cc:1034] running on GCE node
I20260620 14:31:46.535441 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.535682 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.536815 7242 hybrid_clock.cc:648] HybridClock initialized: now 1781965906536802 us; error 28 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.539536 7242 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:31:46.540063 7242 webserver.cc:460] Webserver started at http://127.0.98.1:43547/ using document root <none> and password file <none>
I20260620 14:31:46.540328 7242 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.540403 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.542084 7242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:46.542708 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.542924 7242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:46.543015 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "6c518fa4225d4e768fe62c4ecb6ddd88"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "1dcf694eed30537998fc21d0ee3ba508"
server_key_iv: "59a35328578a555291d0f5cdf7568e4e"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.543859 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.558692 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.560506 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:46.560617 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.560801 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:46.561116 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:46.561148 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.561195 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:46.561223 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.567404 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43379
I20260620 14:31:46.567477 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43379 every 8 connection(s)
I20260620 14:31:46.567832 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:46.570278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7242
I20260620 14:31:46.570451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6694
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:46.578158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40885
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34027
--webserver_port=37287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:31:46.580619 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50391 (local address 127.0.98.61:43929)
0620 14:31:46.568612 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.568614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.568903 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:31:46.569516 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.569517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.569530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.569583 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.570582 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.571279 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.572851 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573138 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573176 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.577120 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.577121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.577121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.577129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.578889 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579402 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.579403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.579403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.579462 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579747 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.579748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.579749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.579906 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.579951 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.580415 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.580503 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:46.581161 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.98.1:50391)
0620 14:31:46.568759 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.568810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.569383 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.569713 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.569714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.570235 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.570487 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.570493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.571436 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.571437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.572754 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.572756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573259 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.573404 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.574216 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.574221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.576996 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.578978 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.578979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.578980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.579306 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.579537 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.579538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.579658 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.580098 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.580099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.580235 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.580708 (+ 473us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.580921 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":55,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260620 14:31:46.581609 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.98.1:40083)
0620 14:31:46.568759 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.568810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.569383 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.569709 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.569714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.570186 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.570487 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.571002 (+ 515us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.572142 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.572143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.572902 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.572903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573389 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.573485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.574147 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.574155 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.576585 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.578841 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.578848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.578851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.579278 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.579513 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.579514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.579658 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.580008 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.580014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.581284 (+ 1270us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.581345 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.581484 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260620 14:31:46.581655 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40593 (local address 127.0.98.62:34027)
0620 14:31:46.569003 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.569005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.569072 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:31:46.569545 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.569546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.569558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.569588 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.570596 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.571418 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.572867 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573209 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573256 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.577607 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.577608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.577609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.577617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.579982 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.580394 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.580396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.580396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.580451 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.580642 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.580642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.580643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.580764 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.580807 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.581506 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.581578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:31:46.581677 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43929
I20260620 14:31:46.581779 7469 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.581990 7469 heartbeater.cc:511] Master 127.0.98.61:43929 requested a full tablet report, sending...
I20260620 14:31:46.582069 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40083 (local address 127.0.98.60:46209)
0620 14:31:46.568864 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.568866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.569551 (+ 685us) server_negotiation.cc:407] Connection header received
0620 14:31:46.569580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.569581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.569594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.569654 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.571090 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.572053 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.572975 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573282 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573340 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.576700 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.576702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.576702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.576710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.578750 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579371 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.579373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.579373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.579439 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.579737 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.579738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.579739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.579910 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.581773 (+ 1863us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.581815 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.581928 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":51,"threads_started":1}
I20260620 14:31:46.582505 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46209
I20260620 14:31:46.582544 7095 ts_manager.cc:194] Registered new tserver with Master: 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379)
I20260620 14:31:46.582564 7467 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.582624 7467 heartbeater.cc:511] Master 127.0.98.60:46209 requested a full tablet report, sending...
I20260620 14:31:46.582963 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.568893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:40593)
0620 14:31:46.569016 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.569025 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.569396 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.569984 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.569985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.570302 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.570489 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.570495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.571530 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.571531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.572752 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.572754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.573333 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.573334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.573404 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.574343 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.574348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.577285 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.580066 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.580067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.580068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.580302 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.580524 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.580524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.580524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.580576 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.581103 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.581104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.581143 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.582470 (+ 1327us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.582576 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":23,"thread_start_us":60,"threads_started":1}
I20260620 14:31:46.582971 7288 ts_manager.cc:194] Registered new tserver with Master: 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379)
I20260620 14:31:46.583294 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:46.583338 7468 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.583395 7468 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:46.583638 6498 ts_manager.cc:194] Re-registered known tserver with Master: 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379)
I20260620 14:31:46.584043 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40593
W20260620 14:31:46.662670 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:46.662879 7475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:46.662904 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:46.664072 7475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:46.664125 7475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:46.664152 7475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:46.665751 7475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37287
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:46.666020 7475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:46.666258 7475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:46.671286 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.671263 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.671459 7475 server_base.cc:1034] running on GCE node
W20260620 14:31:46.671324 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.671743 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.672017 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.673163 7475 hybrid_clock.cc:648] HybridClock initialized: now 1781965906673146 us; error 34 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.675765 7475 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:31:46.676214 7475 webserver.cc:460] Webserver started at http://127.0.98.2:37287/ using document root <none> and password file <none>
I20260620 14:31:46.676455 7475 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.676542 7475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.677948 7475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:31:46.678534 7490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.678712 7475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.678818 7475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "12fa50ee3f2e4d539ee4ca45a05f952a"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "651f80e43d9f5736f7d3c99f51f9c9d5"
server_key_iv: "089def6dd3d50423683cac5268ba3b1f"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.679229 7475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.694965 7475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.697504 7475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:46.697701 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.697980 7475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:46.698295 7475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:46.698349 7475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.698395 7475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:46.698424 7475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.704352 7475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40885
I20260620 14:31:46.704401 7603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40885 every 8 connection(s)
I20260620 14:31:46.704764 7475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:46.705097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7475
I20260620 14:31:46.705268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6828
I20260620 14:31:46.711565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42343
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34027
--webserver_port=34893
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:31:46.718845 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.706178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51851 (local address 127.0.98.60:46209)
0620 14:31:46.706218 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.706220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.706230 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:46.707514 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.707515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.707533 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.707600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.708515 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709390 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711038 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711366 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711413 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.714538 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.714539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.714540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.714547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.716359 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717662 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.717663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.717664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.717735 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.718098 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.718099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.718099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.718236 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.718281 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.718664 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.718750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:46.718845 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.705840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49685 (local address 127.0.98.61:43929)
0620 14:31:46.705887 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.705889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.706585 (+ 696us) server_negotiation.cc:407] Connection header received
0620 14:31:46.708002 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.708003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.708016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.708066 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.708586 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709360 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711033 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711339 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711380 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.715602 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.715604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.715604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.715612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.717135 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717660 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.717661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.717661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.717735 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.718099 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.718100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.718101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.718236 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.718288 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.718664 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.718747 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:31:46.719338 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.705854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:33825)
0620 14:31:46.706608 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.706617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.707990 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.708180 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.708181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.708446 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.708497 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.708498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709524 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.709525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.710923 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.710926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711476 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.711642 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.712331 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.712336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.715001 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.717797 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.717798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.718048 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.718280 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.718280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.718281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.718364 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.718713 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.718714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.718772 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.718854 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.719061 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":84,"thread_start_us":62,"threads_started":1}
I20260620 14:31:46.719336 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.705640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.98.2:49685)
0620 14:31:46.706497 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.706507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.707910 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.708113 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.708114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.708419 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.708490 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.708492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709524 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.709525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.710923 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.710926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711478 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.711642 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.712322 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.712334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.715509 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.717239 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.717241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.717566 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.717806 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.717807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.718014 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.718322 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.718328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.718502 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.719012 (+ 510us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.719137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":155,"thread_start_us":62,"threads_started":1}
I20260620 14:31:46.719661 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.706059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33825 (local address 127.0.98.62:34027)
0620 14:31:46.706105 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.706108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.706661 (+ 553us) server_negotiation.cc:407] Connection header received
0620 14:31:46.708075 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.708077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.708089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.708120 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.708586 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709361 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711033 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711339 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711380 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.715095 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.715098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.715098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.715107 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.717693 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.718141 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.718142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.718143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.718204 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.718462 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.718463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.718464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.718616 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.718684 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.719398 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.719518 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:31:46.720022 7605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:46.720113 7605 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.720291 7605 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:46.720343 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43929
I20260620 14:31:46.720393 7606 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.720485 7606 heartbeater.cc:511] Master 127.0.98.61:43929 requested a full tablet report, sending...
I20260620 14:31:46.720619 6498 ts_manager.cc:194] Re-registered known tserver with Master: 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
I20260620 14:31:46.720795 7095 ts_manager.cc:194] Registered new tserver with Master: 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
I20260620 14:31:46.721097 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33825
I20260620 14:31:46.721252 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.705570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.98.2:51851)
0620 14:31:46.705988 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.706035 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.707393 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.707676 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.707680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.708088 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.708389 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.708395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.709531 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.709532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.710926 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.710929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.711503 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.711504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.711882 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.712486 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.712492 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.714419 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.716441 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.716448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.716451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.717566 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.717812 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.717812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.717813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.718014 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.718328 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.718328 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.718502 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.719300 (+ 798us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.719395 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":114,"thread_start_us":80,"threads_started":1}
I20260620 14:31:46.721679 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46209
I20260620 14:31:46.721740 7604 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.721915 7604 heartbeater.cc:511] Master 127.0.98.60:46209 requested a full tablet report, sending...
I20260620 14:31:46.722234 7288 ts_manager.cc:194] Registered new tserver with Master: 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
W20260620 14:31:46.797469 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:46.797703 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:46.797744 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:46.798952 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:46.799016 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:46.799062 7611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:46.800520 7611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34893
--tserver_master_addrs=127.0.98.62:34027,127.0.98.61:43929,127.0.98.60:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:46.800782 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:46.801023 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:46.806110 7611 server_base.cc:1034] running on GCE node
W20260620 14:31:46.806077 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.806066 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:46.806079 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:46.806550 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:46.806818 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:46.807971 7611 hybrid_clock.cc:648] HybridClock initialized: now 1781965906807958 us; error 43 us; skew 500 ppm
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:46.810599 7611 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:31:46.811025 7611 webserver.cc:460] Webserver started at http://127.0.98.3:34893/ using document root <none> and password file <none>
I20260620 14:31:46.811254 7611 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:46.811333 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:46.812722 7611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:46.813336 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.813514 7611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:46.813627 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
format_stamp: "Formatted at 2026-06-20 14:31:45 on dist-test-slave-tc2g"
server_key: "56eaa369abcbf80e8e884e2b38413d96"
server_key_iv: "90c4d14dc1a79ea4737d4e24343d97c8"
server_key_version: "encryptionkey@0"
I20260620 14:31:46.813975 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:46.832659 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:46.834587 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:46.834755 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:46.834993 7611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:46.835310 7611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:46.835364 7611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.835405 7611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:46.835431 7611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:46.841307 7611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42343
I20260620 14:31:46.841351 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42343 every 8 connection(s)
I20260620 14:31:46.841715 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:46.848340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7611
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:31:46 dist-test-slave-tc2g krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:31:46.855093 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55509 (local address 127.0.98.61:43929)
0620 14:31:46.842478 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.842480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.843089 (+ 609us) server_negotiation.cc:407] Connection header received
0620 14:31:46.843892 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.843893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.843906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.843939 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.844761 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845546 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846456 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.846752 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846818 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.851072 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.851074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.851074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.851082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.853535 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.854015 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.854017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.854017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.854077 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.854328 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.854329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.854329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.854483 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.854558 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.854904 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.854994 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:46.857561 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33057 (local address 127.0.98.60:46209)
0620 14:31:46.842756 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.842758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.842768 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:46.843892 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.843893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.843906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.843939 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.844762 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845540 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846633 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.846897 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846961 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.851230 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.851232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.851232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.851240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.854290 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.855188 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.855189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.855190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.855249 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.855440 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.855441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.855441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.855566 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.855628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.857338 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.857444 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:31:46.858927 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39097 (local address 127.0.98.62:34027)
0620 14:31:46.842981 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.842983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.843099 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:31:46.843871 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.843872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.843884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.843959 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.844745 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845262 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846597 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.846872 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846969 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.851539 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.851541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.851542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.851548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.853567 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.858046 (+ 4479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.858047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.858047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.858112 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.858329 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.858330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.858330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.858456 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.858512 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.858755 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.858827 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:46.859457 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.98.3:33057)
0620 14:31:46.842622 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.842657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.843765 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.844038 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.844043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.844440 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.844654 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.844659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845618 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.845618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.846515 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.846518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.847011 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.847012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.847107 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.847686 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.847696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.851146 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.854654 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.854655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.854655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.855100 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.855321 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.855322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.855322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.855377 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.855655 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.855655 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.855703 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.855776 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.859160 (+ 3384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":3299,"thread_start_us":99,"threads_started":1}
I20260620 14:31:46.859477 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.849894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.98.60:46209)
0620 14:31:46.850959 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.850961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.851066 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:31:46.851333 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.851334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.851345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.851378 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.852235 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.852995 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.854113 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.854407 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.854446 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.856500 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.856501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.856501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.856508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.858174 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.858571 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.858572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.858573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.858629 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.858833 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.858834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.858834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.858989 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.859065 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.859285 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.859372 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":49,"threads_started":1}
I20260620 14:31:46.860059 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46209
I20260620 14:31:46.860170 7740 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.860189 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:39097)
0620 14:31:46.843014 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.843024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.843780 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.844039 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.844043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.844410 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.844652 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.844656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845370 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.845371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.846518 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.846519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.846971 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.847039 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.847754 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.847759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.851036 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.857726 (+ 6690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.857727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.857727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.857964 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.858197 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.858198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.858198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.858252 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.858551 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.858552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.858598 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.858654 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.860077 (+ 1423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1367,"thread_start_us":68,"threads_started":1}
I20260620 14:31:46.860374 7740 heartbeater.cc:511] Master 127.0.98.60:46209 requested a full tablet report, sending...
I20260620 14:31:46.860486 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34027
I20260620 14:31:46.860540 7741 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.860616 7741 heartbeater.cc:511] Master 127.0.98.62:34027 requested a full tablet report, sending...
I20260620 14:31:46.860812 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.842435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.98.3:55509)
0620 14:31:46.843004 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.843014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.843765 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.844047 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.844047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.844459 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:46.844654 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.844659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.845637 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.845638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.846342 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.846344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.846831 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.846832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.846975 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.847700 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:46.847704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:46.850959 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:46.853624 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.853633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.853636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.853923 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.854143 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:46.854143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:46.854144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:46.854264 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:46.854550 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:46.854557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:46.854731 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.854862 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.860691 (+ 5829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260620 14:31:46.860823 6498 ts_manager.cc:194] Re-registered known tserver with Master: fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:46.860972 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.849647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.0.98.62:34027)
0620 14:31:46.849991 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.849994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.850559 (+ 565us) server_negotiation.cc:407] Connection header received
0620 14:31:46.850678 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.850679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.850692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.850734 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.851640 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.852482 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.853661 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.853987 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.854032 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.856244 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.856246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.856246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.856254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.858693 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.859328 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.859329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.859329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.859383 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.859955 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.859956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.859957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.860088 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.860159 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.860766 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.860858 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":84,"threads_started":1}
I20260620 14:31:46.861323 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39097
I20260620 14:31:46.861696 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43929
I20260620 14:31:46.861744 7742 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:46.861817 7742 heartbeater.cc:511] Master 127.0.98.61:43929 requested a full tablet report, sending...
I20260620 14:31:46.861802 7287 ts_manager.cc:194] Registered new tserver with Master: fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:46.862083 7095 ts_manager.cc:194] Registered new tserver with Master: fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:46.862195 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.849812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.61:43929)
0620 14:31:46.851115 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.851117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.851123 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:31:46.851141 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.851141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.851152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.851206 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:46.855168 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.855944 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.857026 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.857523 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.857590 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.859001 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:46.859002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:46.859002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:46.859009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:46.860712 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.861068 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.861069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.861069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.861122 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:46.861287 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:46.861287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:46.861288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:46.861393 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:46.861536 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.861949 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.862029 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":52,"threads_started":1}
I20260620 14:31:46.863838 6498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:46.865490 6498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:46.873059 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43379 (local address 127.0.0.1:55884)
0620 14:31:46.869268 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.869280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.869337 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.869906 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.869907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.869908 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:46.869956 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.869960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.870830 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.870831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.871967 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.871969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.872826 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.872882 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.872885 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.872943 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.872988 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260620 14:31:46.873071 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.0.98.1:43379)
0620 14:31:46.869753 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.869758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.869767 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:46.869802 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.869804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.869816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.870033 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:46.870090 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.870739 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872063 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.872741 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872827 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.872904 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.872960 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.872998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":72,"threads_started":1}
I20260620 14:31:46.873164 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.0.1:47190)
0620 14:31:46.869268 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.869299 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.869340 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.869696 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.869696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.869698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:46.869742 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.869744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.871338 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.871339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.872149 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.872151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.872966 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.873025 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.873029 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.873058 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.873096 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":20,"thread_start_us":66,"threads_started":1}
I20260620 14:31:46.873569 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.98.3:42343)
0620 14:31:46.869473 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.869479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.869489 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:46.869533 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.869535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.869550 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.869640 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:46.869815 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.871279 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872236 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.872869 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.873322 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.873394 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.873453 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.873494 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260620 14:31:46.873579 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.98.2:40885)
0620 14:31:46.869668 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:46.869672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:46.869683 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:46.869720 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:46.869722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:46.869737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:46.869813 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:46.870018 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.871205 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.872525 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.873209 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.873305 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.873380 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:46.873470 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:31:46.873506 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260620 14:31:46.873591 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:46.869138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40885 (local address 127.0.0.1:41738)
0620 14:31:46.869496 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:46.869505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:46.869544 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:46.869879 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:46.869879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:46.869923 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:46.869956 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.869958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.871286 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.871287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.872442 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:46.872444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:46.873306 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:46.873307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:46.873366 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:46.873370 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:46.873472 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:31:46.873518 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":89,"thread_start_us":77,"threads_started":1}
I20260620 14:31:46.874179 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 29acb2c913d745b48920ca914149c9fe (DEFAULT_TABLE table=test-table [id=bb1e4257dbab4d06a260648ddead47f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:46.874454 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29acb2c913d745b48920ca914149c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:46.874349 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 29acb2c913d745b48920ca914149c9fe (DEFAULT_TABLE table=test-table [id=bb1e4257dbab4d06a260648ddead47f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:46.874393 7538 tablet_service.cc:1467] Processing CreateTablet for tablet 29acb2c913d745b48920ca914149c9fe (DEFAULT_TABLE table=test-table [id=bb1e4257dbab4d06a260648ddead47f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:46.874670 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29acb2c913d745b48920ca914149c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:46.874670 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29acb2c913d745b48920ca914149c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:46.878557 7767 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Bootstrap starting.
I20260620 14:31:46.878969 7768 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Bootstrap starting.
I20260620 14:31:46.879094 7766 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Bootstrap starting.
I20260620 14:31:46.879210 7767 tablet_bootstrap.cc:654] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:46.879513 7767 log.cc:824] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:46.879576 7768 tablet_bootstrap.cc:654] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:46.879634 7766 tablet_bootstrap.cc:654] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:46.879926 7766 log.cc:824] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:46.879912 7768 log.cc:824] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:46.880657 7767 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: No bootstrap required, opened a new log
I20260620 14:31:46.880730 7766 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: No bootstrap required, opened a new log
I20260620 14:31:46.880724 7767 ts_tablet_manager.cc:1397] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:46.880790 7766 ts_tablet_manager.cc:1397] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:46.881747 7768 tablet_bootstrap.cc:492] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: No bootstrap required, opened a new log
I20260620 14:31:46.881870 7768 ts_tablet_manager.cc:1397] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:31:46.883082 7767 raft_consensus.cc:348] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.883082 7766 raft_consensus.cc:348] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.883252 7767 raft_consensus.cc:374] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:46.883252 7766 raft_consensus.cc:374] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:46.883294 7766 raft_consensus.cc:729] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c518fa4225d4e768fe62c4ecb6ddd88, State: Initialized, Role: FOLLOWER
I20260620 14:31:46.883294 7767 raft_consensus.cc:729] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf9f56a5ccb4953b626edad28db4c0d, State: Initialized, Role: FOLLOWER
I20260620 14:31:46.883430 7767 consensus_queue.cc:260] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.883437 7766 consensus_queue.cc:260] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.883694 7766 ts_tablet_manager.cc:1428] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:31:46.883708 7767 ts_tablet_manager.cc:1428] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:46.883841 7468 heartbeater.cc:503] Master 127.0.98.62:34027 was elected leader, sending a full tablet report...
I20260620 14:31:46.884076 7768 raft_consensus.cc:348] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.884194 7741 heartbeater.cc:503] Master 127.0.98.62:34027 was elected leader, sending a full tablet report...
I20260620 14:31:46.884241 7768 raft_consensus.cc:374] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:46.884277 7768 raft_consensus.cc:729] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fa50ee3f2e4d539ee4ca45a05f952a, State: Initialized, Role: FOLLOWER
I20260620 14:31:46.884382 7768 consensus_queue.cc:260] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:46.884639 7768 ts_tablet_manager.cc:1428] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:46.884671 7605 heartbeater.cc:503] Master 127.0.98.62:34027 was elected leader, sending a full tablet report...
W20260620 14:31:46.955624 7607 tablet.cc:2369] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:47.017009 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.014371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.0.1:40704)
0620 14:31:47.014431 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:47.014444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:47.014515 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:47.014688 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:47.014689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:47.014690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:47.014750 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.014751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.015408 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.015409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.016122 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.016124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.016754 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.016755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.016815 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.016819 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:47.016861 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:31:47.016914 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:31:47.016995 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.014475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.0.98.60:46209)
0620 14:31:47.014535 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.014538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.014547 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:47.014578 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.014579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.014608 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.014686 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:47.014816 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.015320 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.016216 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.016674 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.016757 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.016858 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.016877 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.016919 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:47.017393 7298 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 14:31:47.017839 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747e8b96132845e0b6277cd90671ed49. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:47.017956 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: LEARNER
I20260620 14:31:47.029086 7774 raft_consensus.cc:482] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:47.029233 7774 raft_consensus.cc:504] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:47.029659 7774 leader_election.cc:288] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379), fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:47.032439 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.029943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52963 (local address 127.0.98.1:43379)
0620 14:31:47.030005 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.030007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.030068 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:31:47.030139 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.030140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.030151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.030228 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:47.030387 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.030835 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.031730 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.032083 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.032173 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.032182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.032338 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.032376 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:31:47.032440 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.029899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42869 (local address 127.0.98.3:42343)
0620 14:31:47.029964 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.029966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.029976 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:31:47.030055 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.030055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.030073 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.030150 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:47.030301 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.030774 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.031736 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.032084 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.032165 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.032175 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.032325 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.032364 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:31:47.032442 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.029816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.2:42869)
0620 14:31:47.029877 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:47.029889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:47.029961 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:47.030155 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:47.030156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:47.030158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:47.030231 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.030233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.030857 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.030858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.031634 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.031636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.032163 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.032164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.032239 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.032242 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:47.032317 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:31:47.032365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":29}
I20260620 14:31:47.032457 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.029839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43379 (local address 127.0.98.2:52963)
0620 14:31:47.030023 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:47.030032 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:47.030075 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:47.030233 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:47.030234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:47.030236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:47.030301 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.030303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.030915 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.030916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.031638 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:47.031640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.032174 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.032175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:47.032259 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.032262 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:47.032327 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:47.032373 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":151}
I20260620 14:31:47.033034 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29acb2c913d745b48920ca914149c9fe" candidate_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" is_pre_election: true
I20260620 14:31:47.033093 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29acb2c913d745b48920ca914149c9fe" candidate_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" is_pre_election: true
I20260620 14:31:47.033212 7694 raft_consensus.cc:2454] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12fa50ee3f2e4d539ee4ca45a05f952a in term 0.
I20260620 14:31:47.033212 7421 raft_consensus.cc:2454] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12fa50ee3f2e4d539ee4ca45a05f952a in term 0.
I20260620 14:31:47.033490 7491 leader_election.cc:302] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fa50ee3f2e4d539ee4ca45a05f952a, fbf9f56a5ccb4953b626edad28db4c0d; no voters:
I20260620 14:31:47.033618 7774 raft_consensus.cc:2781] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:47.033672 7774 raft_consensus.cc:482] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:47.033696 7774 raft_consensus.cc:3037] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:47.034363 7774 raft_consensus.cc:504] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:47.034507 7774 leader_election.cc:288] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [CANDIDATE]: Term 1 election: Requested vote from peers 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379), fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343)
I20260620 14:31:47.034655 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29acb2c913d745b48920ca914149c9fe" candidate_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88"
I20260620 14:31:47.034674 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29acb2c913d745b48920ca914149c9fe" candidate_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
I20260620 14:31:47.034794 7421 raft_consensus.cc:3037] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:47.034788 7694 raft_consensus.cc:3037] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:47.035513 7421 raft_consensus.cc:2454] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12fa50ee3f2e4d539ee4ca45a05f952a in term 1.
I20260620 14:31:47.035723 7493 leader_election.cc:302] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fa50ee3f2e4d539ee4ca45a05f952a, 6c518fa4225d4e768fe62c4ecb6ddd88; no voters:
I20260620 14:31:47.035821 7774 raft_consensus.cc:2781] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:47.035981 7774 raft_consensus.cc:686] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 LEADER]: Becoming Leader. State: Replica: 12fa50ee3f2e4d539ee4ca45a05f952a, State: Running, Role: LEADER
I20260620 14:31:47.036070 7774 consensus_queue.cc:237] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:47.036095 7694 raft_consensus.cc:2454] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12fa50ee3f2e4d539ee4ca45a05f952a in term 1.
I20260620 14:31:47.037086 6498 catalog_manager.cc:5495] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a reported cstate change: term changed from 0 to 1, leader changed from <none> to 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:47.037676 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: true } } }
I20260620 14:31:47.037778 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:47.038537 7252 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: attempting to promote NON_VOTER 9f6a51e8f72d4b9ea8e18fc46b96026c to VOTER
I20260620 14:31:47.038654 7252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } }
I20260620 14:31:47.039041 7297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 LEARNER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 14:31:47.039113 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Refusing update from remote peer 747e8b96132845e0b6277cd90671ed49: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 14:31:47.039307 7252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 14:31:47.039405 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 14:31:47.040019 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040131 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040131 7298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040730 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040799 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:47.040855 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040901 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:47.040943 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.040983 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:47.041050 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.041143 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:47.041311 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "747e8b96132845e0b6277cd90671ed49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:47.041410 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:47.054484 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.051813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.98.2:40885)
0620 14:31:47.051873 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.051875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.051974 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:31:47.052070 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.052071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.052096 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.052185 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:47.052340 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.052821 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.053651 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.053863 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.053955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.054132 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:47.054246 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:47.054334 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.054372 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.054394 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:31:47.056339 7421 raft_consensus.cc:1261] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Refusing update from remote peer 12fa50ee3f2e4d539ee4ca45a05f952a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:47.056352 7694 raft_consensus.cc:1261] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Refusing update from remote peer 12fa50ee3f2e4d539ee4ca45a05f952a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:47.056710 7774 consensus_queue.cc:1035] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:47.056865 7778 consensus_queue.cc:1035] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:47.060947 7783 mvcc.cc:205] Tried to move back new op lower bound from 7298932355299876864 to 7298932355220250624. Current Snapshot: MvccSnapshot[applied={T|T < 7298932355299876864 or (T in {7298932355299876864})}]
I20260620 14:31:47.061012 7785 mvcc.cc:205] Tried to move back new op lower bound from 7298932355299876864 to 7298932355220250624. Current Snapshot: MvccSnapshot[applied={T|T < 7298932355299876864}]
I20260620 14:31:47.061151 7784 mvcc.cc:205] Tried to move back new op lower bound from 7298932355299876864 to 7298932355220250624. Current Snapshot: MvccSnapshot[applied={T|T < 7298932355299876864}]
I20260620 14:31:47.062157 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:47.062255 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:47.063483 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 747e8b96132845e0b6277cd90671ed49: Sending DeleteTablet for 3 replicas of tablet 29acb2c913d745b48920ca914149c9fe
I20260620 14:31:47.064002 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 29acb2c913d745b48920ca914149c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47190
I20260620 14:31:47.064025 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 29acb2c913d745b48920ca914149c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55884
I20260620 14:31:47.064260 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet 29acb2c913d745b48920ca914149c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41738
I20260620 14:31:47.064953 7796 tablet_replica.cc:330] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: stopping tablet replica
I20260620 14:31:47.064999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6466
I20260620 14:31:47.065877 7796 raft_consensus.cc:2229] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:47.066035 7796 raft_consensus.cc:2258] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:47.066582 7796 ts_tablet_manager.cc:1905] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:47.066741 7794 tablet_replica.cc:330] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: stopping tablet replica
I20260620 14:31:47.066877 7794 raft_consensus.cc:2229] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:47.067028 7794 raft_consensus.cc:2258] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:47.069383 7794 ts_tablet_manager.cc:1905] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:47.071283 7794 ts_tablet_manager.cc:1918] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:47.071863 7796 ts_tablet_manager.cc:1918] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:47.072005 7796 log.cc:1196] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/29acb2c913d745b48920ca914149c9fe
I20260620 14:31:47.072403 7796 ts_tablet_manager.cc:1939] T 29acb2c913d745b48920ca914149c9fe P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting consensus metadata
I20260620 14:31:47.072495 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.064283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.98.60:46209)
0620 14:31:47.064319 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.064321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.064330 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:47.064415 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.064416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.064428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.064459 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:47.065249 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.066034 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.067076 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.067355 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.067401 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.069103 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:47.069104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:47.069105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:47.069113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:47.071009 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.071499 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.071500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.071501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.071571 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.071813 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.071815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.071815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.071971 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:47.072053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.072299 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.072388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:47.072772 7794 log.cc:1196] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/29acb2c913d745b48920ca914149c9fe
I20260620 14:31:47.073586 7794 ts_tablet_manager.cc:1939] T 29acb2c913d745b48920ca914149c9fe P fbf9f56a5ccb4953b626edad28db4c0d: Deleting consensus metadata
W20260620 14:31:47.074024 7630 connection.cc:441] server connection from 127.0.0.1:47190 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47190 (request call id 1) could send its response
W20260620 14:31:47.074131 7492 connection.cc:441] server connection from 127.0.0.1:41738 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41738 (request call id 1) could send its response
I20260620 14:31:47.074326 7793 tablet_replica.cc:330] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: stopping tablet replica
I20260620 14:31:47.074460 7793 raft_consensus.cc:2229] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:47.074585 7793 raft_consensus.cc:2258] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:47.075119 7793 ts_tablet_manager.cc:1905] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:47.076539 7793 ts_tablet_manager.cc:1918] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:47.076613 7793 log.cc:1196] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/29acb2c913d745b48920ca914149c9fe
I20260620 14:31:47.076938 7793 ts_tablet_manager.cc:1939] T 29acb2c913d745b48920ca914149c9fe P 6c518fa4225d4e768fe62c4ecb6ddd88: Deleting consensus metadata
W20260620 14:31:47.077306 7354 connection.cc:441] server connection from 127.0.0.1:55884 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55884 (request call id 1) could send its response
I20260620 14:31:47.084492 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.075129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.98.61:43929)
0620 14:31:47.075175 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.075177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.076611 (+ 1434us) server_negotiation.cc:407] Connection header received
0620 14:31:47.076708 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.076709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.076724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.076756 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:47.077477 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.078274 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.079745 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.080062 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.080105 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.081567 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:47.081568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:47.081569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:47.081577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:47.083341 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.083709 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.083711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.083711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.083773 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.083952 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.083953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.083953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.084077 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:47.084180 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.084283 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.084368 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:47.084578 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:47.075018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.60:46209)
0620 14:31:47.075059 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:47.075061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:47.076465 (+ 1404us) server_negotiation.cc:407] Connection header received
0620 14:31:47.076519 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:47.076521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:47.076534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:47.076588 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:47.077552 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.078273 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.079422 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:47.079637 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:47.079724 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:47.081198 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:47.081200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:47.081200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:47.081207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:47.083151 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.083575 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.083576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.083577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.083632 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:47.083987 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:47.083988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:47.083988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:47.084122 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:47.084191 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:47.084401 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:31:47.084479 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:48.061578 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.061264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:43869)
0620 14:31:48.061411 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.061523 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260620 14:31:48.061576 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.061264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:32945)
0620 14:31:48.061441 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.061519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
W20260620 14:31:48.061829 7468 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
W20260620 14:31:48.061877 7605 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.062502 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.062254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:56131)
0620 14:31:48.062376 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.062439 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":50,"threads_started":1}
W20260620 14:31:48.062691 7741 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.564049 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 747e8b96132845e0b6277cd90671ed49)
I20260620 14:31:48.564178 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } }
I20260620 14:31:48.564370 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 747e8b96132845e0b6277cd90671ed49)
I20260620 14:31:48.564461 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027), 16e53c4e15f9461dae98622154f1c21a (127.0.98.61:43929)
I20260620 14:31:48.564762 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } }
I20260620 14:31:48.564858 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.564605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39610)
0620 14:31:48.564730 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.564817 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":46,"threads_started":1}
W20260620 14:31:48.564955 7272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.565302 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.565122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39620)
0620 14:31:48.565201 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.565259 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":49,"threads_started":1}
I20260620 14:31:48.565328 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.565237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39624)
0620 14:31:48.565273 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.565296 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:31:48.565385 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
W20260620 14:31:48.565580 7079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.565589 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027), 9f6a51e8f72d4b9ea8e18fc46b96026c (127.0.98.60:46209)
I20260620 14:31:48.566346 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.565923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39636)
0620 14:31:48.566297 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.566324 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362}
W20260620 14:31:48.566422 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.568358 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.564717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.0.98.61:43929)
0620 14:31:48.564828 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.564831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.565006 (+ 175us) server_negotiation.cc:407] Connection header received
0620 14:31:48.565036 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.565037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.565050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.565105 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.565293 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.566179 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.567544 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.568071 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.568148 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.568161 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.568288 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.568318 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260620 14:31:48.568388 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.564620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43929 (local address 127.0.0.1:42762)
0620 14:31:48.564857 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.564867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.564948 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.565156 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.565157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.565159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.565217 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.565219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.566263 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.566264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.567425 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.567428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.568154 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.568155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.568215 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.568219 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.568277 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.568316 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":35,"threads_started":1}
I20260620 14:31:48.568711 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.565297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46209 (local address 127.0.0.1:40744)
0620 14:31:48.565523 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.565534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.565587 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.565859 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.565860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.565862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.565925 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.565927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.566771 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.566772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.567820 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.567823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.568482 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.568483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.568535 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.568538 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.568562 (+ 24us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.568629 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":39,"threads_started":1}
I20260620 14:31:48.568804 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16e53c4e15f9461dae98622154f1c21a" is_pre_election: true
I20260620 14:31:48.568930 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6a51e8f72d4b9ea8e18fc46b96026c in term 1.
I20260620 14:31:48.569003 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.565489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.60:46209)
0620 14:31:48.565703 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.565704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.565710 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:31:48.565748 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.565748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.565759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.565991 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.566023 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.566689 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.567904 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.568370 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.568852 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.568867 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.568885 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.568930 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260620 14:31:48.569070 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16e53c4e15f9461dae98622154f1c21a, 9f6a51e8f72d4b9ea8e18fc46b96026c; no voters: 747e8b96132845e0b6277cd90671ed49
I20260620 14:31:48.569169 7807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:31:48.569217 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 747e8b96132845e0b6277cd90671ed49)
I20260620 14:31:48.569242 7807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:48.569291 7298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16e53c4e15f9461dae98622154f1c21a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" is_pre_election: true
I20260620 14:31:48.570159 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } }
I20260620 14:31:48.570282 7298 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16e53c4e15f9461dae98622154f1c21a in current term 2: Already voted for candidate 9f6a51e8f72d4b9ea8e18fc46b96026c in this term.
I20260620 14:31:48.570302 7807 leader_election.cc:288] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 election: Requested vote from peers 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027), 16e53c4e15f9461dae98622154f1c21a (127.0.98.61:43929)
I20260620 14:31:48.570482 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.570383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39658)
0620 14:31:48.570429 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.570461 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:31:48.570564 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16e53c4e15f9461dae98622154f1c21a"
I20260620 14:31:48.570598 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16e53c4e15f9461dae98622154f1c21a; no voters: 747e8b96132845e0b6277cd90671ed49, 9f6a51e8f72d4b9ea8e18fc46b96026c
I20260620 14:31:48.570629 7105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:48.570796 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.570709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39674)
0620 14:31:48.570747 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.570771 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:31:48.570842 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
I20260620 14:31:48.571321 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6a51e8f72d4b9ea8e18fc46b96026c in term 2.
I20260620 14:31:48.571403 7808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:31:48.571506 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16e53c4e15f9461dae98622154f1c21a, 9f6a51e8f72d4b9ea8e18fc46b96026c; no voters: 747e8b96132845e0b6277cd90671ed49
I20260620 14:31:48.571635 7807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:31:48.571851 7807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [term 2 LEADER]: Becoming Leader. State: Replica: 9f6a51e8f72d4b9ea8e18fc46b96026c, State: Running, Role: LEADER
I20260620 14:31:48.571928 7807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } }
I20260620 14:31:48.572263 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6a51e8f72d4b9ea8e18fc46b96026c. Latest consensus state: current_term: 2 leader_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:48.572360 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:48.572561 7819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:48.572993 7819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:48.573107 7819 catalog_manager.cc:1259] Loaded cluster ID: a5158a4d5fd440e3bd7ee967a4f7c865
I20260620 14:31:48.573149 7819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:48.573302 7819 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:48.573403 7819 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Loaded TSK: 0
I20260620 14:31:48.573632 7819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:48.622675 7287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:48.624255 7287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:48.625339 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.625190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39702)
0620 14:31:48.625255 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.625292 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:31:48.625344 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [term 2 FOLLOWER]: Refusing update from remote peer 9f6a51e8f72d4b9ea8e18fc46b96026c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 14:31:48.625627 7817 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:31:48.625696 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.625608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.0.1:39708)
0620 14:31:48.625642 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.625674 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 14:31:48.625794 7272 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c -> Peer 747e8b96132845e0b6277cd90671ed49 (127.0.98.62:34027): Couldn't send request to peer 747e8b96132845e0b6277cd90671ed49. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:31:48.626232 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6a51e8f72d4b9ea8e18fc46b96026c. Latest consensus state: current_term: 2 leader_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:48.626335 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:48.626732 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:48.626832 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:48.627059 7821 mvcc.cc:205] Tried to move back new op lower bound from 7298932361727057920 to 7298932361511305216. Current Snapshot: MvccSnapshot[applied={T|T < 7298932361727057920}]
I20260620 14:31:48.627161 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "747e8b96132845e0b6277cd90671ed49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34027 } } peers { permanent_uuid: "16e53c4e15f9461dae98622154f1c21a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43929 } attrs { promote: false } } peers { permanent_uuid: "9f6a51e8f72d4b9ea8e18fc46b96026c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46209 } attrs { promote: false } } }
I20260620 14:31:48.627259 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e53c4e15f9461dae98622154f1c21a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:48.632520 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.0.1:47198)
0620 14:31:48.629208 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.629219 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.629263 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.629761 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.629761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.629763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.629799 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.629800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630761 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.630762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.631540 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.631543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632307 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.632374 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632379 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.632408 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.632447 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":60}
I20260620 14:31:48.632583 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.98.1:43379)
0620 14:31:48.629566 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.629568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.629576 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:48.629605 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.629606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.629618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.629839 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.629878 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630418 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.631751 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632218 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632298 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632311 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.632476 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.632512 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260620 14:31:48.632618 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43379 (local address 127.0.0.1:55888)
0620 14:31:48.629198 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.629208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.629257 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.629711 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.629712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.629716 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.629774 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.629775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630502 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.630504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.631655 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.631656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632307 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.632390 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632397 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.632469 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.632509 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":24}
I20260620 14:31:48.632867 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef7f8aa865146e3baad09357391cb11 (DEFAULT_TABLE table=test-table [id=83a8fbb3a23945dfa734fbaaa37eac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:48.633010 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef7f8aa865146e3baad09357391cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:48.633016 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40885 (local address 127.0.0.1:41762)
0620 14:31:48.629360 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.629371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.629508 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.629866 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.629867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.629870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.629913 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.629914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630761 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.630761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.631865 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.631866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632786 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.632844 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632848 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.632895 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.632937 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260620 14:31:48.633034 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.0.98.3:42343)
0620 14:31:48.629605 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.629608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.629616 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:48.629641 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.629641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.629652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.629716 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.629897 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630659 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.631645 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632213 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632880 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632894 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.632912 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.632963 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260620 14:31:48.633256 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.629479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.98.2:40885)
0620 14:31:48.629707 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.629709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.629716 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:48.629740 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.629741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.629752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.629813 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.629972 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.630688 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.631944 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.632522 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.632590 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.632603 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.633138 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.633193 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260620 14:31:48.633498 7538 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef7f8aa865146e3baad09357391cb11 (DEFAULT_TABLE table=test-table [id=83a8fbb3a23945dfa734fbaaa37eac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:48.633611 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef7f8aa865146e3baad09357391cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:48.634580 7830 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88: Bootstrap starting.
I20260620 14:31:48.634660 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef7f8aa865146e3baad09357391cb11 (DEFAULT_TABLE table=test-table [id=83a8fbb3a23945dfa734fbaaa37eac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:48.634759 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef7f8aa865146e3baad09357391cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:48.634979 7830 tablet_bootstrap.cc:654] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:48.635660 7830 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88: No bootstrap required, opened a new log
I20260620 14:31:48.635725 7830 ts_tablet_manager.cc:1397] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:48.635881 7833 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Bootstrap starting.
I20260620 14:31:48.635938 7831 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Bootstrap starting.
I20260620 14:31:48.635895 7830 raft_consensus.cc:348] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.636040 7830 raft_consensus.cc:374] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:48.636071 7830 raft_consensus.cc:729] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c518fa4225d4e768fe62c4ecb6ddd88, State: Initialized, Role: FOLLOWER
I20260620 14:31:48.636152 7830 consensus_queue.cc:260] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.636252 7833 tablet_bootstrap.cc:654] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:48.636266 7831 tablet_bootstrap.cc:654] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:48.636299 7830 ts_tablet_manager.cc:1428] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:48.636827 7833 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: No bootstrap required, opened a new log
I20260620 14:31:48.636859 7831 tablet_bootstrap.cc:492] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: No bootstrap required, opened a new log
I20260620 14:31:48.636881 7833 ts_tablet_manager.cc:1397] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:48.636900 7831 ts_tablet_manager.cc:1397] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:48.637004 7831 raft_consensus.cc:348] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.637013 7833 raft_consensus.cc:348] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.637080 7833 raft_consensus.cc:374] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:48.637080 7831 raft_consensus.cc:374] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:48.637108 7833 raft_consensus.cc:729] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf9f56a5ccb4953b626edad28db4c0d, State: Initialized, Role: FOLLOWER
I20260620 14:31:48.637108 7831 raft_consensus.cc:729] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fa50ee3f2e4d539ee4ca45a05f952a, State: Initialized, Role: FOLLOWER
I20260620 14:31:48.637151 7833 consensus_queue.cc:260] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.637151 7831 consensus_queue.cc:260] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.637290 7831 ts_tablet_manager.cc:1428] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:48.637295 7833 ts_tablet_manager.cc:1428] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:48.637568 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.636538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:50971)
0620 14:31:48.636660 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.636702 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":82,"threads_started":1}
I20260620 14:31:48.637917 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.637742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:33543)
0620 14:31:48.637840 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.637875 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":68,"threads_started":1}
I20260620 14:31:48.638207 7604 heartbeater.cc:503] Master 127.0.98.60:46209 was elected leader, sending a full tablet report...
I20260620 14:31:48.638262 7740 heartbeater.cc:503] Master 127.0.98.60:46209 was elected leader, sending a full tablet report...
I20260620 14:31:48.638278 7467 heartbeater.cc:503] Master 127.0.98.60:46209 was elected leader, sending a full tablet report...
I20260620 14:31:48.638603 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.638112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:43577)
0620 14:31:48.638335 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.638371 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260620 14:31:48.638862 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.638753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:59633)
0620 14:31:48.638801 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.638831 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:31:48.639164 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.637518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:36531)
0620 14:31:48.639096 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.639131 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":76,"threads_started":1}
I20260620 14:31:48.639483 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.639392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:54807)
0620 14:31:48.639439 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.639463 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:31:48.799932 7834 raft_consensus.cc:482] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:48.800079 7834 raft_consensus.cc:504] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.800500 7834 leader_election.cc:288] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343), 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
I20260620 14:31:48.803408 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.800844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40849 (local address 127.0.98.2:40885)
0620 14:31:48.800890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.800892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.800900 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:48.800928 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.800929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.800947 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.801014 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.801175 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.801695 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.802678 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.803047 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.803128 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.803140 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.803304 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.803337 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:48.803414 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.800695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40885 (local address 127.0.98.1:40849)
0620 14:31:48.800738 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.800750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.800810 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.801020 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.801021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.801024 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.801099 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.801101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.801789 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.801790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.802561 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.802563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.803129 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.803130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.803208 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.803212 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.803292 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.803333 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:31:48.803414 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.800810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49701 (local address 127.0.98.3:42343)
0620 14:31:48.800876 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.800878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.800990 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:31:48.801054 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.801055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.801068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.801144 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:48.801290 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.801821 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.802700 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.803053 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.803136 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.803150 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.803308 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.803342 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:48.803449 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.800712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.1:49701)
0620 14:31:48.800896 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.800907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:48.800994 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:48.801147 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:48.801148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:48.801149 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:48.801195 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.801196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.801902 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.801903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.802606 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:48.802608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.803162 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.803163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:48.803227 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.803229 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:48.803302 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:31:48.803342 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260620 14:31:48.803670 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef7f8aa865146e3baad09357391cb11" candidate_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" is_pre_election: true
I20260620 14:31:48.803670 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef7f8aa865146e3baad09357391cb11" candidate_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" is_pre_election: true
I20260620 14:31:48.803759 7694 raft_consensus.cc:2454] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c518fa4225d4e768fe62c4ecb6ddd88 in term 0.
I20260620 14:31:48.803793 7558 raft_consensus.cc:2454] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c518fa4225d4e768fe62c4ecb6ddd88 in term 0.
I20260620 14:31:48.803962 7356 leader_election.cc:302] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fa50ee3f2e4d539ee4ca45a05f952a, 6c518fa4225d4e768fe62c4ecb6ddd88; no voters:
I20260620 14:31:48.804154 7834 raft_consensus.cc:2781] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:48.804193 7834 raft_consensus.cc:482] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:48.804206 7834 raft_consensus.cc:3037] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:48.804857 7834 raft_consensus.cc:504] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.804999 7834 leader_election.cc:288] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [CANDIDATE]: Term 1 election: Requested vote from peers fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343), 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885)
I20260620 14:31:48.805183 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef7f8aa865146e3baad09357391cb11" candidate_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf9f56a5ccb4953b626edad28db4c0d"
I20260620 14:31:48.805213 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef7f8aa865146e3baad09357391cb11" candidate_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a"
I20260620 14:31:48.805284 7694 raft_consensus.cc:3037] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:48.805281 7558 raft_consensus.cc:3037] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:48.805927 7694 raft_consensus.cc:2454] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c518fa4225d4e768fe62c4ecb6ddd88 in term 1.
I20260620 14:31:48.806078 7558 raft_consensus.cc:2454] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c518fa4225d4e768fe62c4ecb6ddd88 in term 1.
I20260620 14:31:48.806160 7354 leader_election.cc:302] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c518fa4225d4e768fe62c4ecb6ddd88, fbf9f56a5ccb4953b626edad28db4c0d; no voters:
I20260620 14:31:48.806286 7834 raft_consensus.cc:2781] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:48.806479 7834 raft_consensus.cc:686] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [term 1 LEADER]: Becoming Leader. State: Replica: 6c518fa4225d4e768fe62c4ecb6ddd88, State: Running, Role: LEADER
I20260620 14:31:48.806563 7834 consensus_queue.cc:237] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } }
I20260620 14:31:48.806787 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.806677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:41325)
0620 14:31:48.806727 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.806756 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:31:48.807291 7287 catalog_manager.cc:5495] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c518fa4225d4e768fe62c4ecb6ddd88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:48.813446 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.811330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.0.98.1:43379)
0620 14:31:48.811386 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:48.811388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:48.811396 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:48.811445 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:48.811445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:48.811461 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:48.811520 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:48.811619 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.812066 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.812757 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:48.812944 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:48.812999 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:48.813117 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:48.813218 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:48.813268 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:48.813305 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:31:48.813321 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:31:48.814425 7694 raft_consensus.cc:1261] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Refusing update from remote peer 6c518fa4225d4e768fe62c4ecb6ddd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:48.814579 7558 raft_consensus.cc:1261] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Refusing update from remote peer 6c518fa4225d4e768fe62c4ecb6ddd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:48.814692 7844 consensus_queue.cc:1035] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbf9f56a5ccb4953b626edad28db4c0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:48.814872 7834 consensus_queue.cc:1035] T 1ef7f8aa865146e3baad09357391cb11 P 6c518fa4225d4e768fe62c4ecb6ddd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12fa50ee3f2e4d539ee4ca45a05f952a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:48.815788 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.815649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:58799)
0620 14:31:48.815715 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.815753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:31:48.815918 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.815653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:51515)
0620 14:31:48.815859 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.815887 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260620 14:31:48.816983 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.816846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:33867)
0620 14:31:48.816931 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.816957 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 14:31:48.817505 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.817327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.2:44509)
0620 14:31:48.817449 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.817476 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 14:31:48.818584 7287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:48.818646 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.818398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.1:36157)
0620 14:31:48.818595 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.818622 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 14:31:48.818696 7287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:48.819039 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.818816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34027 (local address 127.0.98.3:54753)
0620 14:31:48.818985 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.819012 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 14:31:48.820195 7287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f6a51e8f72d4b9ea8e18fc46b96026c: Sending DeleteTablet for 3 replicas of tablet 1ef7f8aa865146e3baad09357391cb11
I20260620 14:31:48.820456 7674 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef7f8aa865146e3baad09357391cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47198
I20260620 14:31:48.820513 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef7f8aa865146e3baad09357391cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47198
I20260620 14:31:48.820540 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef7f8aa865146e3baad09357391cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55888
I20260620 14:31:48.820683 7538 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef7f8aa865146e3baad09357391cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:48 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41762
I20260620 14:31:48.820883 7858 tablet_replica.cc:330] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: stopping tablet replica
I20260620 14:31:48.820912 7856 tablet_replica.cc:330] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: stopping tablet replica
I20260620 14:31:48.820901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7242
I20260620 14:31:48.820945 7858 raft_consensus.cc:2229] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:48.820971 7856 raft_consensus.cc:2229] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:48.821017 7856 raft_consensus.cc:2258] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:48.821300 7856 ts_tablet_manager.cc:1905] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:48.821483 7858 raft_consensus.cc:2258] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:48.821789 7858 ts_tablet_manager.cc:1905] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:48.823215 7858 ts_tablet_manager.cc:1918] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:48.823297 7858 log.cc:1196] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/1ef7f8aa865146e3baad09357391cb11
I20260620 14:31:48.823328 7856 ts_tablet_manager.cc:1918] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:48.823390 7856 log.cc:1196] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/1ef7f8aa865146e3baad09357391cb11
I20260620 14:31:48.823575 7858 ts_tablet_manager.cc:1939] T 1ef7f8aa865146e3baad09357391cb11 P 12fa50ee3f2e4d539ee4ca45a05f952a: Deleting consensus metadata
I20260620 14:31:48.823619 7856 ts_tablet_manager.cc:1939] T 1ef7f8aa865146e3baad09357391cb11 P fbf9f56a5ccb4953b626edad28db4c0d: Deleting consensus metadata
I20260620 14:31:48.823891 7272 catalog_manager.cc:4841] TS fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343): tablet 1ef7f8aa865146e3baad09357391cb11 (table test-table [id=83a8fbb3a23945dfa734fbaaa37eac9c]) successfully deleted
I20260620 14:31:48.823902 7274 catalog_manager.cc:4841] TS 12fa50ee3f2e4d539ee4ca45a05f952a (127.0.98.2:40885): tablet 1ef7f8aa865146e3baad09357391cb11 (table test-table [id=83a8fbb3a23945dfa734fbaaa37eac9c]) successfully deleted
W20260620 14:31:48.824066 7272 catalog_manager.cc:4808] TS fbf9f56a5ccb4953b626edad28db4c0d (127.0.98.3:42343): delete failed for tablet 1ef7f8aa865146e3baad09357391cb11 because the tablet was not found. No further retry: Not found: Tablet not found: 1ef7f8aa865146e3baad09357391cb11
I20260620 14:31:48.827916 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:48.827839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43379 (local address 127.0.0.1:55920)
0620 14:31:48.827869 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:48.827888 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 14:31:48.828086 7274 catalog_manager.cc:4584] TS 6c518fa4225d4e768fe62c4ecb6ddd88 (127.0.98.1:43379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ef7f8aa865146e3baad09357391cb11: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43379: connect: Connection refused (error 111)
I20260620 14:31:48.828161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7475
I20260620 14:31:48.833978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7611
I20260620 14:31:48.838660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 6962
I20260620 14:31:48.843699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7149
2026-06-20T14:31:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4727 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:48 dist-test-slave-tc2g krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:48 dist-test-slave-tc2g krb5kdc[7861](info): set up 2 sockets
Jun 20 14:31:48 dist-test-slave-tc2g krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260620 14:31:49.933964 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:31:50.051622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:50.131858 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:50.132117 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:50.132154 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:50.133389 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:50.133499 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:31:50.133529 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:50.133563 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:50.133594 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:50.133618 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:50.135371 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:50.135684 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:50.135990 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:50.141430 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.141505 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.141520 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:50.141661 7874 server_base.cc:1034] running on GCE node
I20260620 14:31:50.141922 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:50.142227 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:50.143405 7874 hybrid_clock.cc:648] HybridClock initialized: now 1781965910143392 us; error 30 us; skew 500 ppm
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:50.145838 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:31:50.146317 7874 webserver.cc:460] Webserver started at http://127.0.98.62:34927/ using document root <none> and password file <none>
I20260620 14:31:50.146560 7874 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:50.146626 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:50.146780 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:50.147833 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "33a901f8e71845fdbbc115477dc8399e"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "46a3f8d4291265a49f2b7e9c96f3bc49"
server_key_iv: "f25508c0737101337714a53b1e806f22"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.148196 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "33a901f8e71845fdbbc115477dc8399e"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "46a3f8d4291265a49f2b7e9c96f3bc49"
server_key_iv: "f25508c0737101337714a53b1e806f22"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.149581 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:50.150285 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.150467 7874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:50.150581 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "33a901f8e71845fdbbc115477dc8399e"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "46a3f8d4291265a49f2b7e9c96f3bc49"
server_key_iv: "f25508c0737101337714a53b1e806f22"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.150686 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:50.158418 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:50.160250 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:50.160408 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:50.165092 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36249
I20260620 14:31:50.165131 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36249 every 8 connection(s)
I20260620 14:31:50.165512 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:50.166153 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.166190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7874
I20260620 14:31:50.166316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:50.166518 392 external_mini_cluster.cc:1428] Setting key 6c89d2fe03384f8eb50154b6bcd99663
I20260620 14:31:50.169546 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Bootstrap starting.
I20260620 14:31:50.170214 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.170535 7952 log.cc:824] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:50.171289 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: No bootstrap required, opened a new log
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:50.173750 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:31:50.173890 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.173913 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a901f8e71845fdbbc115477dc8399e, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.174077 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:31:50.174153 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:50.174177 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:50.174238 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:50.174929 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:31:50.175071 7952 leader_election.cc:302] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a901f8e71845fdbbc115477dc8399e; no voters:
I20260620 14:31:50.175246 7952 leader_election.cc:288] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:50.175280 7957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:50.175381 7957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [term 1 LEADER]: Becoming Leader. State: Replica: 33a901f8e71845fdbbc115477dc8399e, State: Running, Role: LEADER
I20260620 14:31:50.175477 7957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:31:50.175518 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:50.175771 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a901f8e71845fdbbc115477dc8399e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 14:31:50.175838 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:50.175953 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a901f8e71845fdbbc115477dc8399e. Latest consensus state: current_term: 1 leader_uuid: "33a901f8e71845fdbbc115477dc8399e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a901f8e71845fdbbc115477dc8399e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 14:31:50.176025 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:50.176843 7961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:50.177366 7961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:50.178020 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.167075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.98.62:36249)
0620 14:31:50.167256 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.167266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.167298 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:31:50.167781 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.167789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.167852 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.168002 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.168569 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.169947 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.170760 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.171319 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.171438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.174071 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.174076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.174077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.174089 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.176184 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.176621 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.176623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.176623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.176732 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.176964 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.176965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.176965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.177370 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.177584 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.177709 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.177846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260620 14:31:50.179353 7961 catalog_manager.cc:1347] Generated new cluster ID: f164ffdbd9b240f7b6fe659e26cef43b
I20260620 14:31:50.179426 7961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:50.197727 7961 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:50.198477 7961 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:50.200829 7961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Generated new TSK 0
I20260620 14:31:50.201025 7961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:31:50.232370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:50.315599 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:50.315814 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:50.315838 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:50.317013 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:50.317055 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:50.317076 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:50.317121 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:50.318614 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:50.318866 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:50.319094 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:50.324082 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.324093 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:50.324286 7978 server_base.cc:1034] running on GCE node
W20260620 14:31:50.324152 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:50.324541 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:50.324735 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:50.325901 7978 hybrid_clock.cc:648] HybridClock initialized: now 1781965910325881 us; error 33 us; skew 500 ppm
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:50.328207 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:31:50.328580 7978 webserver.cc:460] Webserver started at http://127.0.98.1:36689/ using document root <none> and password file <none>
I20260620 14:31:50.328764 7978 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:50.328819 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:50.328915 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:50.329979 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "36b5a2657fdd4522b455083143dd78d3"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "ae229880ed0febbdf0b6900efdec6f7d"
server_key_iv: "96e3cc0ee8d4fcee8fd7d63dc70b1338"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.330274 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36b5a2657fdd4522b455083143dd78d3"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "ae229880ed0febbdf0b6900efdec6f7d"
server_key_iv: "96e3cc0ee8d4fcee8fd7d63dc70b1338"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.331516 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:50.332166 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.332334 7978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:50.332415 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "36b5a2657fdd4522b455083143dd78d3"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "ae229880ed0febbdf0b6900efdec6f7d"
server_key_iv: "96e3cc0ee8d4fcee8fd7d63dc70b1338"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.332468 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:50.342136 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:50.343902 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:50.344023 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:50.344465 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:50.344503 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.344527 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:50.344573 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.351315 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43359
I20260620 14:31:50.351399 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43359 every 8 connection(s)
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:50.351715 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:50.354905 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.344961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58559 (local address 127.0.98.62:36249)
0620 14:31:50.345032 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.345035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.345184 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:31:50.345745 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.345746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.345759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.345838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.346674 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.347556 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.349141 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.349534 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.349621 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.352290 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.352292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.352293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.352301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.353677 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.354054 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.354055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.354055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.354105 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.354372 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.354373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.354373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.354495 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.354551 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.354777 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.354842 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:31:50.355093 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.344912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.1:58559)
0620 14:31:50.345056 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.345096 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.345647 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.345927 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.345931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.346278 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.346567 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.346574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.347653 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.347654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.349039 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.349043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.349622 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.349623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.349723 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.350313 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.350325 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.352171 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.353764 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.353770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.353772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.353991 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.354179 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.354179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.354180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.354299 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.354555 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.354559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.354672 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.354755 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.354871 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260620 14:31:50.355628 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:31:50.355711 8109 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:50.355908 8109 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:31:50.356536 7905 ts_manager.cc:194] Registered new tserver with Master: 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359)
I20260620 14:31:50.357473 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58559
I20260620 14:31:50.357630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 7978
I20260620 14:31:50.357735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:50.357966 392 external_mini_cluster.cc:1428] Setting key 8408b2aac725c197da9cba24d7c64557
I20260620 14:31:50.362879 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.356657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.1:43095)
0620 14:31:50.356788 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.356798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.356854 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.356973 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.356974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.357187 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.357254 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.357256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.358201 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.358202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.358743 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.358745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.359126 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.359127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.359188 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.359660 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.359665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.359946 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.361845 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.361846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.361846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.362012 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.362289 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.362290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.362290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.362351 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.362632 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.362633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.362668 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.362718 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.362801 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260620 14:31:50.362946 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.356701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43095 (local address 127.0.98.62:36249)
0620 14:31:50.356760 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.356762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.356858 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:31:50.356887 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.356887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.356897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.356962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.357318 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.358097 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.358847 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.359053 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.359145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.360057 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.360058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.360059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.360065 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.361739 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.362136 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.362138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.362138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.362203 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.362424 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.362425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.362425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.362563 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.362636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.362748 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.362829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:31:50.383801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:50.464157 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:50.464394 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:50.464432 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:50.465624 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:50.465690 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:50.465725 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:50.465766 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:50.467218 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:50.467478 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:50.467707 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:50.472757 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.472723 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.472702 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:50.472988 8118 server_base.cc:1034] running on GCE node
I20260620 14:31:50.473188 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:50.473500 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:50.474653 8118 hybrid_clock.cc:648] HybridClock initialized: now 1781965910474645 us; error 34 us; skew 500 ppm
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:50.482386 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:31:50.482885 8118 webserver.cc:460] Webserver started at http://127.0.98.2:41825/ using document root <none> and password file <none>
I20260620 14:31:50.483075 8118 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:50.483116 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:50.483207 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:50.484222 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "9803c2cf3a70422d944aaf2869ef7380"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "1daadb0a9f051481835c46426460ced4"
server_key_iv: "dcbffe9e6418e75393cfd0676f859abb"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.484529 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9803c2cf3a70422d944aaf2869ef7380"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "1daadb0a9f051481835c46426460ced4"
server_key_iv: "dcbffe9e6418e75393cfd0676f859abb"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.485898 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:31:50.486689 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.486893 8118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:50.487006 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "9803c2cf3a70422d944aaf2869ef7380"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "1daadb0a9f051481835c46426460ced4"
server_key_iv: "dcbffe9e6418e75393cfd0676f859abb"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.487085 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:50.511296 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:50.513209 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:50.513383 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:50.514009 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:50.514070 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.514111 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:50.514140 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:50.521169 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43373
I20260620 14:31:50.521225 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43373 every 8 connection(s)
I20260620 14:31:50.521644 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:50.524529 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.514427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42539 (local address 127.0.98.62:36249)
0620 14:31:50.514491 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.514493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.514744 (+ 251us) server_negotiation.cc:407] Connection header received
0620 14:31:50.515279 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.515280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.515293 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.515356 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.516080 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.516906 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.518643 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.518881 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.518936 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.521612 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.521614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.521614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.521622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.523138 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.523555 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.523556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.523556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.523621 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.523878 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.523879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.523879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.523997 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.524073 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.524342 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.524427 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:50.524712 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.514424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.2:42539)
0620 14:31:50.514588 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.514624 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.515167 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.515427 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.515431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.515757 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.515953 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.515958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.517023 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.517025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.518541 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.518545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.518976 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.518977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.519153 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.519662 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.519673 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.521521 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.523241 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.523248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.523251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.523475 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.523691 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.523692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.523692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.523804 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.524063 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.524069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.524212 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.524323 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.524487 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260620 14:31:50.525173 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:31:50.525257 8249 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:50.525497 8249 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:31:50.525916 7905 ts_manager.cc:194] Registered new tserver with Master: 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373)
I20260620 14:31:50.526413 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42539
I20260620 14:31:50.528887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8118
I20260620 14:31:50.529011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:50.529214 392 external_mini_cluster.cc:1428] Setting key 3780f120b52f3eaba9766c684e4ae4fe
I20260620 14:31:50.531809 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.526294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34643 (local address 127.0.98.62:36249)
0620 14:31:50.526332 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.526333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.526395 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:31:50.526453 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.526453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.526460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.526546 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.526876 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.527340 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.528047 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.528247 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.528329 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.529270 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.529272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.529272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.529279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.530644 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.531012 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.531013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.531014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.531066 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.531277 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.531278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.531278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.531415 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.531493 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.531648 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.531720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:50.531823 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.526207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.2:34643)
0620 14:31:50.526331 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.526341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.526399 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.526564 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.526565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.526743 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.526798 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.526799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.527431 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.527432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.527962 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.527964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.528331 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.528332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.528394 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.528825 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.528830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.529175 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.530744 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.530745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.530745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.530919 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.531139 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.531140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.531141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.531197 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.531490 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.531490 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.531536 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.531621 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.531721 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:31:50.552299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:50.633074 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:50.633313 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:50.633356 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:50.634603 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:50.634667 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:50.634706 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:50.634761 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:50.636319 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:50.636590 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:50.636814 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:50.641736 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.641716 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:50.641759 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:50.641965 8258 server_base.cc:1034] running on GCE node
I20260620 14:31:50.642143 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:50.642378 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:50.643543 8258 hybrid_clock.cc:648] HybridClock initialized: now 1781965910643515 us; error 39 us; skew 500 ppm
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:50.646008 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:31:50.646487 8258 webserver.cc:460] Webserver started at http://127.0.98.3:45465/ using document root <none> and password file <none>
I20260620 14:31:50.646723 8258 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:50.646792 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:50.646939 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:50.647964 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "de7cac1330a8454987e8b483fa83db9d"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "c0ac608e0622ea2723827d9ddab3c052"
server_key_iv: "42b69914ef0f97b1760b8a8f8a7453aa"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.648315 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de7cac1330a8454987e8b483fa83db9d"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "c0ac608e0622ea2723827d9ddab3c052"
server_key_iv: "42b69914ef0f97b1760b8a8f8a7453aa"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.649698 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:31:50.650430 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.650635 8258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:50.650732 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "de7cac1330a8454987e8b483fa83db9d"
format_stamp: "Formatted at 2026-06-20 14:31:50 on dist-test-slave-tc2g"
server_key: "c0ac608e0622ea2723827d9ddab3c052"
server_key_iv: "42b69914ef0f97b1760b8a8f8a7453aa"
server_key_version: "encryptionkey@0"
I20260620 14:31:50.650811 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:50.676347 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:50.678953 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:50.679123 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:50.679708 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:50.679767 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:50.679807 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:50.679827 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:50.686955 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41683
I20260620 14:31:50.687031 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41683 every 8 connection(s)
I20260620 14:31:50.687374 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:50.689260 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.680090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51167 (local address 127.0.98.62:36249)
0620 14:31:50.680138 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.680141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.680392 (+ 251us) server_negotiation.cc:407] Connection header received
0620 14:31:50.680871 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.680872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.680885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.680982 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.681697 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.682208 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.683297 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.683550 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.683637 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.686446 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.686448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.686448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.686457 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.688066 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.688485 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.688486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.688487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.688536 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.688738 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.688739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.688739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.688851 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.688903 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.689138 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.689199 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:50.689419 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.680094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.3:51167)
0620 14:31:50.680250 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.680292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.680776 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.681049 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.681052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.681340 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.681600 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.681605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.682303 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.682304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.683211 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.683214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.683646 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.683647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.683774 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.684304 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.684312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.686335 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.688137 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.688141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.688143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.688421 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.688594 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.688594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.688595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.688686 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.688905 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.688908 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.689034 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.689115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.689230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260620 14:31:50.689939 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:31:50.690022 8389 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:50.690218 8389 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:31:50.690611 7906 ts_manager.cc:194] Registered new tserver with Master: de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:50.691138 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51167
I20260620 14:31:50.696197 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.691027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41913 (local address 127.0.98.62:36249)
0620 14:31:50.691064 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.691065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.691109 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:31:50.691177 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.691177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.691185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.691257 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.691623 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.692082 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.692779 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.692984 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.693066 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.693968 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.693970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.693970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.693977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.695073 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.695423 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.695424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.695425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.695497 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.695689 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.695690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.695690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.695801 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.695879 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.696027 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.696102 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:31:50.696210 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.690931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.3:41913)
0620 14:31:50.691055 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.691065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.691122 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.691276 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.691277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.691473 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:50.691533 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.691535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.692170 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.692170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.692696 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.692697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.693068 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.693069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.693135 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.693612 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:50.693616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:50.693881 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:50.695172 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.695174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.695174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.695339 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.695562 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:50.695562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:50.695563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:50.695619 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:50.695866 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:50.695868 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:50.695910 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.696001 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.696103 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 14:31:50.697490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8258
I20260620 14:31:50.697582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:50.697724 392 external_mini_cluster.cc:1428] Setting key ea864aa42c08c00d09a857b7f099ea78
I20260620 14:31:50.698483 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:50.718555 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:50.713626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.0.98.62:36249)
0620 14:31:50.713683 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.713687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.713700 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:50.713725 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.713725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.713733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.713784 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.714299 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.714774 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.715446 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.715618 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.715665 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.718470 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59134: BlockingRecv error: recv got EOF from 127.0.0.1:59134 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:31:50 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:50.725531 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.719324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.0.98.62:36249)
0620 14:31:50.719359 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.719361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.719425 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:31:50.719480 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.719480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.719487 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.719535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:50.719951 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.720411 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.721131 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.721304 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.721353 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.723392 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:50.723393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:50.723393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:50.723398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:50.724541 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.724825 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.724826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.724826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.724871 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:50.725021 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:50.725021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:50.725022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:50.725168 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:50.725240 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.725359 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.725413 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:50.727166 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:50.728746 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:50.735447 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43373 (local address 127.0.0.1:33002)
0620 14:31:50.731681 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.731691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.731736 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.731994 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.731995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.731996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:50.732042 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.732043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.733068 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.733069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.734250 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.734252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.735151 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.735152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.735214 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.735226 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.735312 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.735363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":7,"thread_start_us":90,"threads_started":1}
I20260620 14:31:50.735473 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.98.2:43373)
0620 14:31:50.731424 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.731428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.731790 (+ 362us) server_negotiation.cc:407] Connection header received
0620 14:31:50.731832 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.731834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.731851 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.731928 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:50.732123 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.732978 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.734340 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.735072 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.735152 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.735234 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.735328 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.735369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260620 14:31:50.735893 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43359 (local address 127.0.0.1:44378)
0620 14:31:50.731480 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.731488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.731562 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.732318 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.732319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.732321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:50.732361 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.732362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.733332 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.733333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.734264 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.734266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.735670 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.735671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.735730 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.735734 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.735798 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.735835 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":91,"threads_started":1}
I20260620 14:31:50.735962 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.0.98.1:43359)
0620 14:31:50.731424 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.731428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.731557 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:31:50.732204 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.732207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.732224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.732310 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:50.732421 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.733252 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.734380 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.735083 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.735152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.735231 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.735818 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.735861 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260620 14:31:50.736248 8183 tablet_service.cc:1467] Processing CreateTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 (DEFAULT_TABLE table=test-table [id=090e90c1a7fb441c98c5d653b500a60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.736510 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41683 (local address 127.0.0.1:47370)
0620 14:31:50.731151 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.731192 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.731253 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.731590 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.731593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.731595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:50.731672 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.731674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.732851 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.732852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.734249 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.734252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.736251 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.736252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.736333 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.736337 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.736396 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.736432 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260620 14:31:50.736548 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.731085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.98.3:41683)
0620 14:31:50.731416 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.731422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.731433 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:50.731471 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.731473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.731488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.731585 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:50.731754 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.732769 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.735269 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.736170 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.736249 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.736330 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.736412 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.736448 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260620 14:31:50.736663 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92f32bd56ba4e3c8162a592b6e67d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.736555 8043 tablet_service.cc:1467] Processing CreateTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 (DEFAULT_TABLE table=test-table [id=090e90c1a7fb441c98c5d653b500a60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.736868 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92f32bd56ba4e3c8162a592b6e67d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.739912 8417 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Bootstrap starting.
I20260620 14:31:50.740512 8418 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Bootstrap starting.
I20260620 14:31:50.740617 8417 tablet_bootstrap.cc:654] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.740609 8323 tablet_service.cc:1467] Processing CreateTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 (DEFAULT_TABLE table=test-table [id=090e90c1a7fb441c98c5d653b500a60a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.740973 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92f32bd56ba4e3c8162a592b6e67d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.740958 8417 log.cc:824] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:50.741143 8418 tablet_bootstrap.cc:654] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.741483 8418 log.cc:824] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:50.741812 8417 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: No bootstrap required, opened a new log
I20260620 14:31:50.741897 8417 ts_tablet_manager.cc:1397] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:50.742241 8418 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: No bootstrap required, opened a new log
I20260620 14:31:50.742303 8418 ts_tablet_manager.cc:1397] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:50.743983 8421 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Bootstrap starting.
I20260620 14:31:50.743990 8417 raft_consensus.cc:348] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.744179 8417 raft_consensus.cc:374] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.744221 8417 raft_consensus.cc:729] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.744364 8417 consensus_queue.cc:260] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.744306 8418 raft_consensus.cc:348] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.744448 8418 raft_consensus.cc:374] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.744482 8418 raft_consensus.cc:729] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.744645 8417 ts_tablet_manager.cc:1428] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:50.744616 8418 consensus_queue.cc:260] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.744772 8249 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
I20260620 14:31:50.744869 8418 ts_tablet_manager.cc:1428] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:50.744876 8421 tablet_bootstrap.cc:654] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.745018 8109 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
I20260620 14:31:50.745198 8421 log.cc:824] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:50.745967 8421 tablet_bootstrap.cc:492] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: No bootstrap required, opened a new log
I20260620 14:31:50.746038 8421 ts_tablet_manager.cc:1397] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:50.747334 8421 raft_consensus.cc:348] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.747453 8421 raft_consensus.cc:374] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.747476 8421 raft_consensus.cc:729] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cac1330a8454987e8b483fa83db9d, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.747609 8421 consensus_queue.cc:260] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.747859 8421 ts_tablet_manager.cc:1428] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:50.747920 8389 heartbeater.cc:503] Master 127.0.98.62:36249 was elected leader, sending a full tablet report...
W20260620 14:31:50.772271 8250 tablet.cc:2369] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:50.852510 8110 tablet.cc:2369] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:50.895256 8423 raft_consensus.cc:482] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:50.895394 8423 raft_consensus.cc:504] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.895848 8423 leader_election.cc:288] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683), 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373)
I20260620 14:31:50.898787 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.896189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43295 (local address 127.0.98.2:43373)
0620 14:31:50.896233 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.896236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.896260 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:31:50.896320 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.896321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.896336 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.896392 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:50.896564 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.897061 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898061 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.898439 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898494 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.898504 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.898668 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.898706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:31:50.898797 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.896049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43373 (local address 127.0.98.1:43295)
0620 14:31:50.896200 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.896212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.896257 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.896407 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.896407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.896410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:50.896482 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.896486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.897162 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.897163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.897956 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.897958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.898515 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.898587 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.898591 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.898644 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.898689 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":16,"thread_start_us":60,"threads_started":1}
I20260620 14:31:50.898882 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.896159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41503 (local address 127.0.98.3:41683)
0620 14:31:50.896216 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.896219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.896233 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:50.896312 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.896313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.896324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.896399 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:50.896574 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.897042 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898133 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.898517 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898587 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.898596 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.898771 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.898812 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:31:50.898945 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.896036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41683 (local address 127.0.98.1:41503)
0620 14:31:50.896174 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:50.896186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:50.896247 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:50.896403 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:50.896404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:50.896406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:50.896479 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.896481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.897118 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.897119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.898039 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:50.898042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.898600 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.898601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:50.898692 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.898696 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:50.898756 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:31:50.898795 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":61,"thread_start_us":91,"threads_started":1}
I20260620 14:31:50.899302 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92f32bd56ba4e3c8162a592b6e67d02" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d" is_pre_election: true
I20260620 14:31:50.899358 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92f32bd56ba4e3c8162a592b6e67d02" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803c2cf3a70422d944aaf2869ef7380" is_pre_election: true
I20260620 14:31:50.899457 8343 raft_consensus.cc:2454] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 0.
I20260620 14:31:50.899537 8203 raft_consensus.cc:2454] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 0.
I20260620 14:31:50.899693 7994 leader_election.cc:302] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b5a2657fdd4522b455083143dd78d3, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:50.899840 8423 raft_consensus.cc:2781] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:50.899927 8423 raft_consensus.cc:482] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:50.899964 8423 raft_consensus.cc:3037] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:50.900665 8423 raft_consensus.cc:504] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.900821 8423 leader_election.cc:288] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 election: Requested vote from peers de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683), 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373)
I20260620 14:31:50.900956 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92f32bd56ba4e3c8162a592b6e67d02" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d"
I20260620 14:31:50.901010 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92f32bd56ba4e3c8162a592b6e67d02" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803c2cf3a70422d944aaf2869ef7380"
I20260620 14:31:50.901057 8343 raft_consensus.cc:3037] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:50.901111 8203 raft_consensus.cc:3037] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:50.901859 8343 raft_consensus.cc:2454] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 1.
I20260620 14:31:50.901870 8203 raft_consensus.cc:2454] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 1.
I20260620 14:31:50.902062 7997 leader_election.cc:302] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b5a2657fdd4522b455083143dd78d3, 9803c2cf3a70422d944aaf2869ef7380; no voters:
I20260620 14:31:50.902175 8423 raft_consensus.cc:2781] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:50.902307 8423 raft_consensus.cc:686] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 1 LEADER]: Becoming Leader. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Running, Role: LEADER
I20260620 14:31:50.902427 8423 consensus_queue.cc:237] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } }
I20260620 14:31:50.903256 7906 catalog_manager.cc:5495] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "36b5a2657fdd4522b455083143dd78d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:50.917533 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:50.915189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.0.98.1:43359)
0620 14:31:50.915249 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:50.915251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:50.915262 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:50.915305 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:50.915306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:50.915329 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:50.915409 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:50.915530 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.915994 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.916747 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:50.916939 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:50.917011 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:50.917200 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:50.917299 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:50.917391 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:50.917410 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:31:50.917430 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:31:50.919591 8343 raft_consensus.cc:1261] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Refusing update from remote peer 36b5a2657fdd4522b455083143dd78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:50.919610 8203 raft_consensus.cc:1261] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Refusing update from remote peer 36b5a2657fdd4522b455083143dd78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:50.919909 8423 consensus_queue.cc:1035] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:50.920038 8428 consensus_queue.cc:1035] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:50.922494 8431 mvcc.cc:205] Tried to move back new op lower bound from 7298932371123470336 to 7298932371056955392. Current Snapshot: MvccSnapshot[applied={T|T < 7298932371123470336}]
I20260620 14:31:50.923204 8433 mvcc.cc:205] Tried to move back new op lower bound from 7298932371123470336 to 7298932371056955392. Current Snapshot: MvccSnapshot[applied={T|T < 7298932371123470336}]
I20260620 14:31:50.925663 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:50.925750 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:50.926491 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Sending DeleteTablet for 3 replicas of tablet f92f32bd56ba4e3c8162a592b6e67d02
I20260620 14:31:50.926795 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47370
I20260620 14:31:50.926820 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44378
I20260620 14:31:50.926822 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet f92f32bd56ba4e3c8162a592b6e67d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 14:31:50.927213 8442 tablet_replica.cc:330] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: stopping tablet replica
I20260620 14:31:50.927289 8442 raft_consensus.cc:2229] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:50.927363 8442 raft_consensus.cc:2258] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:50.927371 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:50.927502 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:50.927733 8442 ts_tablet_manager.cc:1905] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:50.928941 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b (DEFAULT_TABLE table=test-table [id=47ea0de7b28548aab739a8e19065b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.929005 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b (DEFAULT_TABLE table=test-table [id=47ea0de7b28548aab739a8e19065b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.929085 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4c3b56945b4eda8b08c8a9cef9822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.929117 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4c3b56945b4eda8b08c8a9cef9822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.929224 8442 ts_tablet_manager.cc:1918] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:50.929282 8442 log.cc:1196] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/f92f32bd56ba4e3c8162a592b6e67d02
I20260620 14:31:50.929558 8442 ts_tablet_manager.cc:1939] T f92f32bd56ba4e3c8162a592b6e67d02 P 9803c2cf3a70422d944aaf2869ef7380: Deleting consensus metadata
I20260620 14:31:50.930351 8421 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Bootstrap starting.
I20260620 14:31:50.930418 8418 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Bootstrap starting.
I20260620 14:31:50.930814 8418 tablet_bootstrap.cc:654] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.930896 8421 tablet_bootstrap.cc:654] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.931232 7893 catalog_manager.cc:4841] TS 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373): tablet f92f32bd56ba4e3c8162a592b6e67d02 (table test-table [id=090e90c1a7fb441c98c5d653b500a60a]) successfully deleted
I20260620 14:31:50.931331 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b (DEFAULT_TABLE table=test-table [id=47ea0de7b28548aab739a8e19065b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:50.931437 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4c3b56945b4eda8b08c8a9cef9822b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:50.932602 8417 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Bootstrap starting.
I20260620 14:31:50.932972 8417 tablet_bootstrap.cc:654] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:50.933219 8418 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: No bootstrap required, opened a new log
I20260620 14:31:50.933274 8418 ts_tablet_manager.cc:1397] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:50.933447 8418 raft_consensus.cc:348] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.933526 8418 raft_consensus.cc:374] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.933557 8418 raft_consensus.cc:729] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.933606 8418 consensus_queue.cc:260] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.933629 8441 tablet_replica.cc:330] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: stopping tablet replica
I20260620 14:31:50.933704 8418 ts_tablet_manager.cc:1428] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:50.933746 8441 raft_consensus.cc:2229] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:50.933780 8417 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: No bootstrap required, opened a new log
I20260620 14:31:50.933815 8417 ts_tablet_manager.cc:1397] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:50.933864 8441 raft_consensus.cc:2258] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:50.933558 8440 tablet_replica.cc:330] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: stopping tablet replica
I20260620 14:31:50.933951 8417 raft_consensus.cc:348] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.934043 8417 raft_consensus.cc:374] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.934088 8417 raft_consensus.cc:729] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.934142 8417 consensus_queue.cc:260] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.934233 8417 ts_tablet_manager.cc:1428] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:50.934333 8441 ts_tablet_manager.cc:1905] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:50.934517 8440 raft_consensus.cc:2229] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:50.934664 8440 raft_consensus.cc:2258] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:50.935104 8440 ts_tablet_manager.cc:1905] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:50.935829 8441 ts_tablet_manager.cc:1918] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:50.935901 8441 log.cc:1196] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/f92f32bd56ba4e3c8162a592b6e67d02
I20260620 14:31:50.936156 8441 ts_tablet_manager.cc:1939] T f92f32bd56ba4e3c8162a592b6e67d02 P de7cac1330a8454987e8b483fa83db9d: Deleting consensus metadata
I20260620 14:31:50.936614 8440 ts_tablet_manager.cc:1918] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:50.936606 7890 catalog_manager.cc:4841] TS de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683): tablet f92f32bd56ba4e3c8162a592b6e67d02 (table test-table [id=090e90c1a7fb441c98c5d653b500a60a]) successfully deleted
I20260620 14:31:50.936681 8440 log.cc:1196] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/f92f32bd56ba4e3c8162a592b6e67d02
I20260620 14:31:50.936905 8440 ts_tablet_manager.cc:1939] T f92f32bd56ba4e3c8162a592b6e67d02 P 36b5a2657fdd4522b455083143dd78d3: Deleting consensus metadata
I20260620 14:31:50.937336 7892 catalog_manager.cc:4841] TS 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359): tablet f92f32bd56ba4e3c8162a592b6e67d02 (table test-table [id=090e90c1a7fb441c98c5d653b500a60a]) successfully deleted
I20260620 14:31:50.941834 8421 tablet_bootstrap.cc:492] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: No bootstrap required, opened a new log
I20260620 14:31:50.941898 8421 ts_tablet_manager.cc:1397] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260620 14:31:50.942126 8421 raft_consensus.cc:348] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.942206 8421 raft_consensus.cc:374] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:50.942230 8421 raft_consensus.cc:729] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cac1330a8454987e8b483fa83db9d, State: Initialized, Role: FOLLOWER
I20260620 14:31:50.942284 8421 consensus_queue.cc:260] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:50.942397 8421 ts_tablet_manager.cc:1428] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:51.090937 8422 raft_consensus.cc:482] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:51.091059 8422 raft_consensus.cc:504] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.091507 8422 leader_election.cc:288] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.094465 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.091882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59715 (local address 127.0.98.1:43359)
0620 14:31:51.091971 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.091973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.091981 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:51.092015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.092016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.092028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.092099 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:51.092266 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.092746 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.093734 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.094103 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.094186 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.094197 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.094347 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.094386 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:31:51.094475 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.091816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51537 (local address 127.0.98.3:41683)
0620 14:31:51.091867 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.091870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.092043 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:31:51.092107 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.092108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.092117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.092191 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:51.092333 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.092790 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.093768 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.094129 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.094211 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.094220 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.094368 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.094405 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:51.094520 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.091727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41683 (local address 127.0.98.2:51537)
0620 14:31:51.091942 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.091955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:51.092048 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:51.092195 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:51.092195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:51.092197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:51.092252 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.092254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.092875 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.092876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.093680 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.093682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.094212 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.094213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.094291 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.094294 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:31:51.094361 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:51.094403 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260620 14:31:51.094519 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.091727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43359 (local address 127.0.98.2:59715)
0620 14:31:51.091850 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.091862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:51.091924 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:51.092106 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:51.092107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:51.092111 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:51.092193 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.092195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.092832 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.092833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.093638 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.093640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.094188 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.094189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.094271 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.094275 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:51.094342 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:51.094384 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260620 14:31:51.095005 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4c3b56945b4eda8b08c8a9cef9822b" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d" is_pre_election: true
I20260620 14:31:51.094975 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4c3b56945b4eda8b08c8a9cef9822b" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3" is_pre_election: true
I20260620 14:31:51.095094 8343 raft_consensus.cc:2454] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.095172 8063 raft_consensus.cc:2454] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.095314 8134 leader_election.cc:302] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9803c2cf3a70422d944aaf2869ef7380, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:51.095463 8422 raft_consensus.cc:2781] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:51.095515 8422 raft_consensus.cc:482] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:51.095531 8422 raft_consensus.cc:3037] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.096230 8422 raft_consensus.cc:504] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.096376 8422 leader_election.cc:288] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Requested vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.096524 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4c3b56945b4eda8b08c8a9cef9822b" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d"
I20260620 14:31:51.096598 8343 raft_consensus.cc:3037] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.097275 8343 raft_consensus.cc:2454] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.097368 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4c3b56945b4eda8b08c8a9cef9822b" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3"
I20260620 14:31:51.097453 8134 leader_election.cc:302] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9803c2cf3a70422d944aaf2869ef7380, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:51.097489 8063 raft_consensus.cc:3037] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.097599 8422 raft_consensus.cc:2781] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:51.097757 8422 raft_consensus.cc:686] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 1 LEADER]: Becoming Leader. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Running, Role: LEADER
I20260620 14:31:51.097831 8422 consensus_queue.cc:237] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.098356 8063 raft_consensus.cc:2454] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.098567 7905 catalog_manager.cc:5495] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:51.145030 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.138329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.0.98.62:36249)
0620 14:31:51.138391 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.138393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.138538 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:31:51.138585 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.138586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.138595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.138646 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:51.139056 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.139522 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.140232 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.140420 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.140485 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.142600 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:51.142601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:51.142601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:51.142607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:51.143808 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:51.144220 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:51.144220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:51.144221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:51.144274 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:51.144481 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:51.144482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:51.144482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:51.144588 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:51.144662 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.144819 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.144950 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":35}
I20260620 14:31:51.145030 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.138290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.0.1:59160)
0620 14:31:51.138456 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.138466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:51.138542 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:51.138663 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:51.138664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:51.138951 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:51.138996 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.138998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.139584 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.139584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.140131 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.140133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.140491 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.140492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.140535 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.141050 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:51.141055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:51.142524 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:51.143926 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:51.143933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:51.143935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:51.144131 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:51.144342 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:51.144343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:51.144343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:51.144417 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:51.144654 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:51.144660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:51.144706 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:31:51.144787 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:31:51.144904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260620 14:31:51.147137 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:31:51.147354 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:51.148768 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 277b9b58790d415290d06e4a4d7758cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a64b3a48455c4268b20438d06d348930]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:51.148768 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 277b9b58790d415290d06e4a4d7758cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a64b3a48455c4268b20438d06d348930]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:51.148798 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 277b9b58790d415290d06e4a4d7758cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a64b3a48455c4268b20438d06d348930]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:51.148947 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277b9b58790d415290d06e4a4d7758cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.148947 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277b9b58790d415290d06e4a4d7758cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.148963 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277b9b58790d415290d06e4a4d7758cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.150254 8421 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: Bootstrap starting.
I20260620 14:31:51.150411 8417 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: Bootstrap starting.
I20260620 14:31:51.150419 8418 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: Bootstrap starting.
I20260620 14:31:51.150719 8418 tablet_bootstrap.cc:654] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.150719 8421 tablet_bootstrap.cc:654] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.150748 8417 tablet_bootstrap.cc:654] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.151335 8418 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: No bootstrap required, opened a new log
I20260620 14:31:51.151400 8418 ts_tablet_manager.cc:1397] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:51.151458 8417 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: No bootstrap required, opened a new log
I20260620 14:31:51.151510 8417 ts_tablet_manager.cc:1397] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:51.151552 8418 raft_consensus.cc:348] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.151628 8418 raft_consensus.cc:374] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.151659 8418 raft_consensus.cc:729] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.151676 8417 raft_consensus.cc:348] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.151711 8418 consensus_queue.cc:260] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.151746 8417 raft_consensus.cc:374] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.151767 8417 raft_consensus.cc:729] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.151798 8418 ts_tablet_manager.cc:1428] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:51.151815 8417 consensus_queue.cc:260] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.151885 8423 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.151913 8417 ts_tablet_manager.cc:1428] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:51.151969 8423 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.152101 8422 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.152176 8422 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.152717 8421 tablet_bootstrap.cc:492] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: No bootstrap required, opened a new log
I20260620 14:31:51.152789 8421 ts_tablet_manager.cc:1397] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:51.152932 8421 raft_consensus.cc:348] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.152985 8421 raft_consensus.cc:374] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.153002 8421 raft_consensus.cc:729] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cac1330a8454987e8b483fa83db9d, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.153054 8421 consensus_queue.cc:260] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.153147 8421 ts_tablet_manager.cc:1428] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.153198 8425 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.153285 8425 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:31:51.188092 8390 tablet.cc:2369] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:51.273537 8422 raft_consensus.cc:482] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:51.273622 8422 raft_consensus.cc:504] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.273792 8422 leader_election.cc:288] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.274098 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277b9b58790d415290d06e4a4d7758cd" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3" is_pre_election: true
I20260620 14:31:51.274098 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277b9b58790d415290d06e4a4d7758cd" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d" is_pre_election: true
I20260620 14:31:51.274211 8343 raft_consensus.cc:2454] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.274211 8063 raft_consensus.cc:2454] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.274381 8134 leader_election.cc:302] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9803c2cf3a70422d944aaf2869ef7380, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:51.274482 8422 raft_consensus.cc:2781] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:51.274564 8422 raft_consensus.cc:482] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:51.274595 8422 raft_consensus.cc:3037] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.275342 8422 raft_consensus.cc:504] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.275488 8422 leader_election.cc:288] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Requested vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.275646 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277b9b58790d415290d06e4a4d7758cd" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d"
I20260620 14:31:51.275686 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277b9b58790d415290d06e4a4d7758cd" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3"
I20260620 14:31:51.275751 8343 raft_consensus.cc:3037] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.275758 8063 raft_consensus.cc:3037] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.276428 8063 raft_consensus.cc:2454] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.276525 8343 raft_consensus.cc:2454] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.276611 8136 leader_election.cc:302] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b5a2657fdd4522b455083143dd78d3, 9803c2cf3a70422d944aaf2869ef7380; no voters:
I20260620 14:31:51.276696 8422 raft_consensus.cc:2781] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:51.276778 8422 raft_consensus.cc:686] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [term 1 LEADER]: Becoming Leader. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Running, Role: LEADER
I20260620 14:31:51.276832 8422 consensus_queue.cc:237] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.277129 8448 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: TxnStatusTablet state changed. Reason: New leader 9803c2cf3a70422d944aaf2869ef7380. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.277232 8448 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:51.277536 8460 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:31:51.277649 8460 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:31:51.277670 7905 catalog_manager.cc:5495] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:51.301638 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.298696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43373 (local address 127.0.0.1:33014)
0620 14:31:51.298795 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.298808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:51.298864 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:51.299064 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:51.299064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:51.299065 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:51.299109 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.299111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.300019 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.300020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.300587 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.300589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.301066 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.301067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.301117 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.301132 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.301478 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.301485 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:51.301534 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:31:51.301578 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:31:51.301649 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.298768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.0.98.2:43373)
0620 14:31:51.298821 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.298823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.298838 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:31:51.298956 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.298957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.298979 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.299054 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:51.299152 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.299926 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.300674 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.300984 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.301058 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.301224 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.301366 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.301477 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.301541 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.301565 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:31:51.302827 8343 raft_consensus.cc:1261] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.302935 8063 raft_consensus.cc:1261] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.303269 8448 consensus_queue.cc:1035] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.303426 8422 consensus_queue.cc:1035] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.304064 8425 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: TxnStatusTablet state changed. Reason: New leader 9803c2cf3a70422d944aaf2869ef7380. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.304195 8425 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.304175 8423 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: TxnStatusTablet state changed. Reason: New leader 9803c2cf3a70422d944aaf2869ef7380. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.304280 8423 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.305095 8422 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.305179 8422 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:51.305135 8448 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.305217 8448 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 9803c2cf3a70422d944aaf2869ef7380: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:51.305590 8423 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.305673 8423 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P 36b5a2657fdd4522b455083143dd78d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.305732 8432 mvcc.cc:205] Tried to move back new op lower bound from 7298932372694327296 to 7298932372590190592. Current Snapshot: MvccSnapshot[applied={T|T < 7298932372694327296 or (T in {7298932372694327296})}]
I20260620 14:31:51.305958 8425 tablet_replica.cc:439] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } } }
I20260620 14:31:51.306041 8425 tablet_replica.cc:442] T 277b9b58790d415290d06e4a4d7758cd P de7cac1330a8454987e8b483fa83db9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:51.309615 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.307051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.0.98.2:43373)
0620 14:31:51.307087 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.307089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.307153 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:31:51.307225 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.307226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.307234 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.307272 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:51.307402 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.307853 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.308961 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.309148 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.309199 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.309344 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.309368 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.309400 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.309502 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.309518 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:31:51.314747 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.312299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34239 (local address 127.0.98.2:43373)
0620 14:31:51.312341 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.312343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.312508 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:31:51.312565 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.312566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.312575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.312628 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:51.312767 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.313212 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.314046 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.314218 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.314296 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.314469 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.314502 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.314574 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.314651 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.314666 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:31:51.314810 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.312242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43373 (local address 127.0.98.2:34239)
0620 14:31:51.312440 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.312449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:51.312517 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:51.312630 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:51.312631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:51.312632 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:51.312695 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.312696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.313304 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.313305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.313968 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:51.313970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.314297 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.314298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:51.314362 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.314382 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.314576 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.314578 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:31:51.314646 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:31:51.314688 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260620 14:31:51.317241 8343 raft_consensus.cc:1261] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.317276 8063 raft_consensus.cc:1261] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.317533 8465 consensus_queue.cc:1035] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.317648 8422 consensus_queue.cc:1035] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.331936 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.332074 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.332739 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Sending DeleteTablet for 3 replicas of tablet 1f4c3b56945b4eda8b08c8a9cef9822b
I20260620 14:31:51.332993 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 14:31:51.333032 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47370
I20260620 14:31:51.333093 8442 tablet_replica.cc:330] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: stopping tablet replica
I20260620 14:31:51.333141 8442 raft_consensus.cc:2229] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:51.333139 8441 tablet_replica.cc:330] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: stopping tablet replica
I20260620 14:31:51.333145 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4c3b56945b4eda8b08c8a9cef9822b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44378
I20260620 14:31:51.333192 8441 raft_consensus.cc:2229] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.333225 8440 tablet_replica.cc:330] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: stopping tablet replica
I20260620 14:31:51.333228 8441 raft_consensus.cc:2258] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.333240 8442 raft_consensus.cc:2258] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.333273 8440 raft_consensus.cc:2229] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.333312 8440 raft_consensus.cc:2258] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.333729 8440 ts_tablet_manager.cc:1905] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.333729 8441 ts_tablet_manager.cc:1905] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.335255 8440 ts_tablet_manager.cc:1918] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:51.335255 8441 ts_tablet_manager.cc:1918] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:51.335319 8441 log.cc:1196] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/1f4c3b56945b4eda8b08c8a9cef9822b
I20260620 14:31:51.335319 8440 log.cc:1196] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/1f4c3b56945b4eda8b08c8a9cef9822b
I20260620 14:31:51.335546 8441 ts_tablet_manager.cc:1939] T 1f4c3b56945b4eda8b08c8a9cef9822b P de7cac1330a8454987e8b483fa83db9d: Deleting consensus metadata
I20260620 14:31:51.335546 8440 ts_tablet_manager.cc:1939] T 1f4c3b56945b4eda8b08c8a9cef9822b P 36b5a2657fdd4522b455083143dd78d3: Deleting consensus metadata
I20260620 14:31:51.335789 8442 ts_tablet_manager.cc:1905] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.335932 7892 catalog_manager.cc:4841] TS 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359): tablet 1f4c3b56945b4eda8b08c8a9cef9822b (table test-table [id=47ea0de7b28548aab739a8e19065b488]) successfully deleted
I20260620 14:31:51.336205 7890 catalog_manager.cc:4841] TS de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683): tablet 1f4c3b56945b4eda8b08c8a9cef9822b (table test-table [id=47ea0de7b28548aab739a8e19065b488]) successfully deleted
I20260620 14:31:51.337317 8442 ts_tablet_manager.cc:1918] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:51.337375 8442 log.cc:1196] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/1f4c3b56945b4eda8b08c8a9cef9822b
I20260620 14:31:51.337613 8442 ts_tablet_manager.cc:1939] T 1f4c3b56945b4eda8b08c8a9cef9822b P 9803c2cf3a70422d944aaf2869ef7380: Deleting consensus metadata
I20260620 14:31:51.337904 7893 catalog_manager.cc:4841] TS 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373): tablet 1f4c3b56945b4eda8b08c8a9cef9822b (table test-table [id=47ea0de7b28548aab739a8e19065b488]) successfully deleted
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:51.357949 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:51.351929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.0.98.62:36249)
0620 14:31:51.351988 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.351990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.352110 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:31:51.352191 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.352192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.352203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.352278 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:51.352856 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.353679 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.354606 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.354821 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.354893 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.357878 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59166: BlockingRecv error: recv got EOF from 127.0.0.1:59166 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:31:51.366483 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.358815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.98.62:36249)
0620 14:31:51.358857 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.358859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.359119 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:31:51.359152 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.359153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.359163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.359243 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:51.359857 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.360306 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.361260 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.361474 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.361539 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.363859 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:51.363861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:51.363861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:51.363869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:51.365309 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:51.365704 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:51.365705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:51.365706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:51.365765 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:51.365965 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:51.365966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:51.365966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:51.366089 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:51.366216 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.366298 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.366376 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:31:51.368165 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:51.368335 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:51.369740 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa (DEFAULT_TABLE table=test-table [id=cdec67752d33435798fdbc43970937d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.369731 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa (DEFAULT_TABLE table=test-table [id=cdec67752d33435798fdbc43970937d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.369817 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa (DEFAULT_TABLE table=test-table [id=cdec67752d33435798fdbc43970937d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.369864 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4b977b84b345bc85133e0c2962bbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.369865 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4b977b84b345bc85133e0c2962bbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.369947 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4b977b84b345bc85133e0c2962bbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.371227 8418 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Bootstrap starting.
I20260620 14:31:51.371304 8421 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Bootstrap starting.
I20260620 14:31:51.371554 8418 tablet_bootstrap.cc:654] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.371733 8421 tablet_bootstrap.cc:654] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.371826 8417 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Bootstrap starting.
I20260620 14:31:51.372262 8417 tablet_bootstrap.cc:654] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.372434 8421 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: No bootstrap required, opened a new log
I20260620 14:31:51.372488 8421 ts_tablet_manager.cc:1397] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:51.372642 8421 raft_consensus.cc:348] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.372723 8421 raft_consensus.cc:374] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.372752 8421 raft_consensus.cc:729] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cac1330a8454987e8b483fa83db9d, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.372794 8421 consensus_queue.cc:260] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.372885 8421 ts_tablet_manager.cc:1428] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.373010 8418 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: No bootstrap required, opened a new log
I20260620 14:31:51.373065 8418 ts_tablet_manager.cc:1397] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:51.373175 8418 raft_consensus.cc:348] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.373242 8418 raft_consensus.cc:374] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.373261 8418 raft_consensus.cc:729] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.373297 8418 consensus_queue.cc:260] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.373625 8417 tablet_bootstrap.cc:492] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: No bootstrap required, opened a new log
I20260620 14:31:51.373675 8417 ts_tablet_manager.cc:1397] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:51.373821 8417 raft_consensus.cc:348] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.373893 8417 raft_consensus.cc:374] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.373910 8417 raft_consensus.cc:729] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.373953 8417 consensus_queue.cc:260] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.374047 8417 ts_tablet_manager.cc:1428] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:51.373406 8418 ts_tablet_manager.cc:1428] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.491973 8423 raft_consensus.cc:482] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:51.492084 8423 raft_consensus.cc:504] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.492244 8423 leader_election.cc:288] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.492451 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4b977b84b345bc85133e0c2962bbaa" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803c2cf3a70422d944aaf2869ef7380" is_pre_election: true
I20260620 14:31:51.492472 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4b977b84b345bc85133e0c2962bbaa" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d" is_pre_election: true
I20260620 14:31:51.492569 8343 raft_consensus.cc:2454] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 0.
I20260620 14:31:51.492571 8203 raft_consensus.cc:2454] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 0.
I20260620 14:31:51.492730 7997 leader_election.cc:302] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b5a2657fdd4522b455083143dd78d3, 9803c2cf3a70422d944aaf2869ef7380; no voters:
I20260620 14:31:51.492822 8423 raft_consensus.cc:2781] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:51.492858 8423 raft_consensus.cc:482] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:51.492877 8423 raft_consensus.cc:3037] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.493757 8423 raft_consensus.cc:504] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.493892 8423 leader_election.cc:288] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 election: Requested vote from peers 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.494042 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4b977b84b345bc85133e0c2962bbaa" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803c2cf3a70422d944aaf2869ef7380"
I20260620 14:31:51.494112 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4b977b84b345bc85133e0c2962bbaa" candidate_uuid: "36b5a2657fdd4522b455083143dd78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d"
I20260620 14:31:51.494148 8203 raft_consensus.cc:3037] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.494189 8343 raft_consensus.cc:3037] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.494742 8203 raft_consensus.cc:2454] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 1.
I20260620 14:31:51.494925 7997 leader_election.cc:302] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36b5a2657fdd4522b455083143dd78d3, 9803c2cf3a70422d944aaf2869ef7380; no voters:
I20260620 14:31:51.494951 8343 raft_consensus.cc:2454] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b5a2657fdd4522b455083143dd78d3 in term 1.
I20260620 14:31:51.495041 8423 raft_consensus.cc:2781] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:51.495142 8423 raft_consensus.cc:686] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 1 LEADER]: Becoming Leader. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Running, Role: LEADER
I20260620 14:31:51.495208 8423 consensus_queue.cc:237] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.495848 7905 catalog_manager.cc:5495] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "36b5a2657fdd4522b455083143dd78d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:51.521848 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.519222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.98.1:43359)
0620 14:31:51.519295 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.519297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.519308 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:51.519334 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.519335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.519350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.519416 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:51.519537 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.520205 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.521005 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.521277 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.521356 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.521580 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.521615 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.521674 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.521748 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.521773 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:31:51.522923 8343 raft_consensus.cc:1261] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Refusing update from remote peer 36b5a2657fdd4522b455083143dd78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.523350 8423 consensus_queue.cc:1035] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.524950 8203 raft_consensus.cc:1261] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Refusing update from remote peer 36b5a2657fdd4522b455083143dd78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.525360 8488 consensus_queue.cc:1035] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.526213 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.526304 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.527060 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Sending DeleteTablet for 3 replicas of tablet 0c4b977b84b345bc85133e0c2962bbaa
I20260620 14:31:51.527302 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 14:31:51.527344 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44378
I20260620 14:31:51.527400 8442 tablet_replica.cc:330] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: stopping tablet replica
I20260620 14:31:51.527428 8440 tablet_replica.cc:330] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: stopping tablet replica
I20260620 14:31:51.527484 8440 raft_consensus.cc:2229] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:51.527485 8442 raft_consensus.cc:2229] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.527529 8442 raft_consensus.cc:2258] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.527540 8440 raft_consensus.cc:2258] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.527590 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47370
I20260620 14:31:51.527745 8441 tablet_replica.cc:330] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: stopping tablet replica
I20260620 14:31:51.527803 8441 raft_consensus.cc:2229] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.527833 8442 ts_tablet_manager.cc:1905] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.527860 8441 raft_consensus.cc:2258] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.527387 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4b977b84b345bc85133e0c2962bbaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 14:31:51.528228 8440 ts_tablet_manager.cc:1905] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.528285 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:51.528414 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:51.529458 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e0c17785f491199e2a160d7611e06 (DEFAULT_TABLE table=test-table [id=e0b2fe0685214ec5afd3e01821137bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.529484 8442 ts_tablet_manager.cc:1918] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:51.529541 8442 log.cc:1196] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/0c4b977b84b345bc85133e0c2962bbaa
I20260620 14:31:51.529569 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e0c17785f491199e2a160d7611e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.529762 8442 ts_tablet_manager.cc:1939] T 0c4b977b84b345bc85133e0c2962bbaa P 9803c2cf3a70422d944aaf2869ef7380: Deleting consensus metadata
I20260620 14:31:51.529829 8440 ts_tablet_manager.cc:1918] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:51.529874 8440 log.cc:1196] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/0c4b977b84b345bc85133e0c2962bbaa
I20260620 14:31:51.530086 8440 ts_tablet_manager.cc:1939] T 0c4b977b84b345bc85133e0c2962bbaa P 36b5a2657fdd4522b455083143dd78d3: Deleting consensus metadata
I20260620 14:31:51.530543 8441 ts_tablet_manager.cc:1905] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.530609 8417 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Bootstrap starting.
I20260620 14:31:51.530709 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e0c17785f491199e2a160d7611e06 (DEFAULT_TABLE table=test-table [id=e0b2fe0685214ec5afd3e01821137bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.530793 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e0c17785f491199e2a160d7611e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.530910 8417 tablet_bootstrap.cc:654] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.531725 8441 ts_tablet_manager.cc:1918] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:51.531795 8441 log.cc:1196] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/0c4b977b84b345bc85133e0c2962bbaa
I20260620 14:31:51.531813 8418 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: Bootstrap starting.
I20260620 14:31:51.531808 7892 catalog_manager.cc:4841] TS 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359): tablet 0c4b977b84b345bc85133e0c2962bbaa (table test-table [id=cdec67752d33435798fdbc43970937d5]) successfully deleted
I20260620 14:31:51.531984 7893 catalog_manager.cc:4841] TS 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373): tablet 0c4b977b84b345bc85133e0c2962bbaa (table test-table [id=cdec67752d33435798fdbc43970937d5]) successfully deleted
I20260620 14:31:51.532001 8441 ts_tablet_manager.cc:1939] T 0c4b977b84b345bc85133e0c2962bbaa P de7cac1330a8454987e8b483fa83db9d: Deleting consensus metadata
W20260620 14:31:51.532086 7893 catalog_manager.cc:4808] TS 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373): delete failed for tablet 0c4b977b84b345bc85133e0c2962bbaa because the tablet was not found. No further retry: Not found: Tablet not found: 0c4b977b84b345bc85133e0c2962bbaa
I20260620 14:31:51.532127 8418 tablet_bootstrap.cc:654] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.532366 7890 catalog_manager.cc:4841] TS de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683): tablet 0c4b977b84b345bc85133e0c2962bbaa (table test-table [id=cdec67752d33435798fdbc43970937d5]) successfully deleted
I20260620 14:31:51.532449 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8e0c17785f491199e2a160d7611e06 (DEFAULT_TABLE table=test-table [id=e0b2fe0685214ec5afd3e01821137bda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:51.532538 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8e0c17785f491199e2a160d7611e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:51.532701 8418 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: No bootstrap required, opened a new log
I20260620 14:31:51.532735 8418 ts_tablet_manager.cc:1397] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:51.532823 8418 raft_consensus.cc:348] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.532876 8418 raft_consensus.cc:374] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.532894 8418 raft_consensus.cc:729] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b5a2657fdd4522b455083143dd78d3, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.532935 8418 consensus_queue.cc:260] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.532980 8417 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: No bootstrap required, opened a new log
I20260620 14:31:51.533015 8418 ts_tablet_manager.cc:1428] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.533021 8417 ts_tablet_manager.cc:1397] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:51.533113 8417 raft_consensus.cc:348] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.533170 8417 raft_consensus.cc:374] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.533195 8417 raft_consensus.cc:729] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.533237 8417 consensus_queue.cc:260] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.533306 8417 ts_tablet_manager.cc:1428] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.533820 8421 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Bootstrap starting.
I20260620 14:31:51.534318 8421 tablet_bootstrap.cc:654] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:51.534919 8421 tablet_bootstrap.cc:492] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: No bootstrap required, opened a new log
I20260620 14:31:51.534967 8421 ts_tablet_manager.cc:1397] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:51.535102 8421 raft_consensus.cc:348] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.535169 8421 raft_consensus.cc:374] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:51.535188 8421 raft_consensus.cc:729] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cac1330a8454987e8b483fa83db9d, State: Initialized, Role: FOLLOWER
I20260620 14:31:51.535228 8421 consensus_queue.cc:260] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.535326 8421 ts_tablet_manager.cc:1428] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:51.659292 8471 raft_consensus.cc:482] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:51.659404 8471 raft_consensus.cc:504] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.659730 8471 leader_election.cc:288] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.659968 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e0c17785f491199e2a160d7611e06" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d" is_pre_election: true
I20260620 14:31:51.659986 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e0c17785f491199e2a160d7611e06" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3" is_pre_election: true
I20260620 14:31:51.660059 8343 raft_consensus.cc:2454] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.660090 8063 raft_consensus.cc:2454] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 0.
I20260620 14:31:51.660216 8134 leader_election.cc:302] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9803c2cf3a70422d944aaf2869ef7380, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:51.660327 8471 raft_consensus.cc:2781] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:51.660383 8471 raft_consensus.cc:482] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:51.660400 8471 raft_consensus.cc:3037] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.661119 8471 raft_consensus.cc:504] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.661275 8471 leader_election.cc:288] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Requested vote from peers 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359), de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683)
I20260620 14:31:51.661453 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e0c17785f491199e2a160d7611e06" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b5a2657fdd4522b455083143dd78d3"
I20260620 14:31:51.661492 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8e0c17785f491199e2a160d7611e06" candidate_uuid: "9803c2cf3a70422d944aaf2869ef7380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7cac1330a8454987e8b483fa83db9d"
I20260620 14:31:51.661561 8063 raft_consensus.cc:3037] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.661558 8343 raft_consensus.cc:3037] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:51.662158 8343 raft_consensus.cc:2454] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.662330 8063 raft_consensus.cc:2454] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9803c2cf3a70422d944aaf2869ef7380 in term 1.
I20260620 14:31:51.662340 8134 leader_election.cc:302] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9803c2cf3a70422d944aaf2869ef7380, de7cac1330a8454987e8b483fa83db9d; no voters:
I20260620 14:31:51.662423 8471 raft_consensus.cc:2781] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:51.662460 8471 raft_consensus.cc:686] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 LEADER]: Becoming Leader. State: Replica: 9803c2cf3a70422d944aaf2869ef7380, State: Running, Role: LEADER
I20260620 14:31:51.662511 8471 consensus_queue.cc:237] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } }
I20260620 14:31:51.663151 7905 catalog_manager.cc:5495] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9803c2cf3a70422d944aaf2869ef7380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9803c2cf3a70422d944aaf2869ef7380" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:51.683735 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.681786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.98.2:43373)
0620 14:31:51.681825 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:51.681827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:51.681857 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:31:51.681910 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:51.681911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:51.681921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:51.681965 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:51.682079 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.682538 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.683190 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:51.683360 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:51.683409 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:51.683521 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:51.683544 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:51.683577 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:51.683624 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:31:51.683637 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:31:51.685743 8343 raft_consensus.cc:1261] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.685751 8063 raft_consensus.cc:1261] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Refusing update from remote peer 9803c2cf3a70422d944aaf2869ef7380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:51.685971 8471 consensus_queue.cc:1035] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7cac1330a8454987e8b483fa83db9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.686084 8465 consensus_queue.cc:1035] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b5a2657fdd4522b455083143dd78d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:51.698689 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.698809 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:51.699436 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a901f8e71845fdbbc115477dc8399e: Sending DeleteTablet for 3 replicas of tablet 4c8e0c17785f491199e2a160d7611e06
I20260620 14:31:51.699688 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8e0c17785f491199e2a160d7611e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44378
I20260620 14:31:51.699687 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8e0c17785f491199e2a160d7611e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47370
I20260620 14:31:51.699805 8441 tablet_replica.cc:330] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: stopping tablet replica
I20260620 14:31:51.699819 8440 tablet_replica.cc:330] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: stopping tablet replica
I20260620 14:31:51.699873 8441 raft_consensus.cc:2229] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.699878 8440 raft_consensus.cc:2229] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:51.699915 8441 raft_consensus.cc:2258] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.699915 8440 raft_consensus.cc:2258] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.699990 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8e0c17785f491199e2a160d7611e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33002
I20260620 14:31:51.700067 8442 tablet_replica.cc:330] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: stopping tablet replica
I20260620 14:31:51.700109 8442 raft_consensus.cc:2229] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:51.700161 8442 raft_consensus.cc:2258] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:51.700196 8440 ts_tablet_manager.cc:1905] T 4c8e0c17785f491199e2a160d7611e06 P 36b5a2657fdd4522b455083143dd78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.700258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7978
I20260620 14:31:51.700479 8442 ts_tablet_manager.cc:1905] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.700500 8441 ts_tablet_manager.cc:1905] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:51.701962 8442 ts_tablet_manager.cc:1918] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:51.702044 8442 log.cc:1196] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/4c8e0c17785f491199e2a160d7611e06
I20260620 14:31:51.702265 8442 ts_tablet_manager.cc:1939] T 4c8e0c17785f491199e2a160d7611e06 P 9803c2cf3a70422d944aaf2869ef7380: Deleting consensus metadata
I20260620 14:31:51.702310 8441 ts_tablet_manager.cc:1918] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:51.702375 8441 log.cc:1196] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/4c8e0c17785f491199e2a160d7611e06
I20260620 14:31:51.702566 7893 catalog_manager.cc:4841] TS 9803c2cf3a70422d944aaf2869ef7380 (127.0.98.2:43373): tablet 4c8e0c17785f491199e2a160d7611e06 (table test-table [id=e0b2fe0685214ec5afd3e01821137bda]) successfully deleted
I20260620 14:31:51.702600 8441 ts_tablet_manager.cc:1939] T 4c8e0c17785f491199e2a160d7611e06 P de7cac1330a8454987e8b483fa83db9d: Deleting consensus metadata
I20260620 14:31:51.702899 7890 catalog_manager.cc:4841] TS de7cac1330a8454987e8b483fa83db9d (127.0.98.3:41683): tablet 4c8e0c17785f491199e2a160d7611e06 (table test-table [id=e0b2fe0685214ec5afd3e01821137bda]) successfully deleted
W20260620 14:31:51.708189 7892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43359 (error 108)
I20260620 14:31:51.708398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8118
I20260620 14:31:51.709048 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:51.708538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43359 (local address 127.0.0.1:49832)
0620 14:31:51.708963 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:51.709002 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":35,"threads_started":1}
W20260620 14:31:51.709136 7892 catalog_manager.cc:4584] TS 36b5a2657fdd4522b455083143dd78d3 (127.0.98.1:43359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c8e0c17785f491199e2a160d7611e06: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43359: connect: Connection refused (error 111)
I20260620 14:31:51.714646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8258
I20260620 14:31:51.719478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 7874
2026-06-20T14:31:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2874 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[8503](info): set up 2 sockets
Jun 20 14:31:51 dist-test-slave-tc2g krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260620 14:31:52.802162 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:52 dist-test-slave-tc2g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:52.917415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:52.997929 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:52.998137 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:52.998158 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:52.999341 8516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:52.999385 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:52.999399 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:52.999411 8516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:52.999464 8516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:53.001001 8516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:53.001255 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:53.001531 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:53.006240 8516 server_base.cc:1034] running on GCE node
W20260620 14:31:53.006207 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:53.006263 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:53.006203 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:53.006685 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:53.007009 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:53.008172 8516 hybrid_clock.cc:648] HybridClock initialized: now 1781965913008158 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:31:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1294 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:53 dist-test-slave-tc2g krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:53 dist-test-slave-tc2g krb5kdc[8529](info): set up 2 sockets
Jun 20 14:31:53 dist-test-slave-tc2g krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260620 14:31:54.050159 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:31:54.163636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:54.245620 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:54.245819 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:54.245841 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:54.247028 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:54.247071 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:54.247097 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:54.247139 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:54.247165 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:54.248752 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:54.249007 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:54.249197 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:54.253835 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.254034 8542 server_base.cc:1034] running on GCE node
W20260620 14:31:54.253870 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.253868 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.254377 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:54.254685 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:54.255863 8542 hybrid_clock.cc:648] HybridClock initialized: now 1781965914255852 us; error 32 us; skew 500 ppm
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:54.258438 8542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:54.258904 8542 webserver.cc:460] Webserver started at http://127.0.98.62:39849/ using document root <none> and password file <none>
I20260620 14:31:54.259158 8542 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:54.259238 8542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:54.259372 8542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:54.260403 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "43c4a3262e6242979f77a9bad7963f65"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "a763ce5bf169dee5f6b7bab666c8b115"
server_key_iv: "4d045ba74fac4aea682125301cfdbbac"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.260758 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "43c4a3262e6242979f77a9bad7963f65"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "a763ce5bf169dee5f6b7bab666c8b115"
server_key_iv: "4d045ba74fac4aea682125301cfdbbac"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.262060 8542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:31:54.262755 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.262913 8542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:54.263012 8542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "43c4a3262e6242979f77a9bad7963f65"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "a763ce5bf169dee5f6b7bab666c8b115"
server_key_iv: "4d045ba74fac4aea682125301cfdbbac"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.263095 8542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:54.272598 8542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:54.274911 8542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:54.275080 8542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:54.279157 8542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42463
I20260620 14:31:54.279207 8609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42463 every 8 connection(s)
I20260620 14:31:54.279744 8542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:54.280398 8610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:54.282610 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Bootstrap starting.
I20260620 14:31:54.283129 8610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:54.283358 8610 log.cc:824] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:54.283965 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: No bootstrap required, opened a new log
I20260620 14:31:54.285465 8610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:54.285599 8610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:54.285621 8610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c4a3262e6242979f77a9bad7963f65, State: Initialized, Role: FOLLOWER
I20260620 14:31:54.285719 8610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:54.285796 8610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:54.285831 8610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:54.285879 8610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:54.286839 8610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:54.286960 8610 leader_election.cc:302] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c4a3262e6242979f77a9bad7963f65; no voters:
I20260620 14:31:54.287135 8610 leader_election.cc:288] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:54.287173 8613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:54.287364 8613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 LEADER]: Becoming Leader. State: Replica: 43c4a3262e6242979f77a9bad7963f65, State: Running, Role: LEADER
I20260620 14:31:54.287398 8610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:54.287454 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:54.287865 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c4a3262e6242979f77a9bad7963f65. Latest consensus state: current_term: 1 leader_uuid: "43c4a3262e6242979f77a9bad7963f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260620 14:31:54.287959 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:54.288165 8622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:54.288579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8542
I20260620 14:31:54.288697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:31:54.288707 8622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:54.288909 392 external_mini_cluster.cc:1428] Setting key 8d49e471db43f4cfdc9d909c4ce29b3f
I20260620 14:31:54.289147 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c4a3262e6242979f77a9bad7963f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260620 14:31:54.289499 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:54.290283 8622 catalog_manager.cc:1347] Generated new cluster ID: 96dea574aa7742f7ab402158bf9091ec
I20260620 14:31:54.290338 8622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:54.299885 8622 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:54.300578 8622 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:54.300854 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.289487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.0.98.62:42463)
0620 14:31:54.289799 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:54.289808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:54.289839 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:31:54.289954 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:54.289961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:54.290018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:54.290983 (+ 965us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:54.291089 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.292535 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.293607 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.294257 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.294367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.297150 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:54.297154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:54.297154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:54.297166 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:54.299196 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.299612 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.299616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.299616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.299724 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.299916 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.299917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.299918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.300182 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:54.300327 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:54.300465 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:31:54.300631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":75,"threads_started":1}
I20260620 14:31:54.312721 8622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Generated new TSK 0
I20260620 14:31:54.312950 8622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:31:54.342023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:54.422524 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:54.422760 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:54.422798 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:54.423938 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:54.424003 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:54.424046 8636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:54.425566 8636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:54.425828 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:54.426046 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:54.430986 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.431007 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.431160 8636 server_base.cc:1034] running on GCE node
W20260620 14:31:54.431007 8644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.431429 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:54.431675 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:54.432818 8636 hybrid_clock.cc:648] HybridClock initialized: now 1781965914432805 us; error 30 us; skew 500 ppm
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:54.435443 8636 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:31:54.435930 8636 webserver.cc:460] Webserver started at http://127.0.98.1:34173/ using document root <none> and password file <none>
I20260620 14:31:54.436160 8636 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:54.436226 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:54.436359 8636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:54.437536 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "20318447a752469eb0ba4e766821ab1e"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "c136fcf1602e6bac485a92907782fe5d"
server_key_iv: "af114c7eb71078c5abca080e4a126411"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.437887 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20318447a752469eb0ba4e766821ab1e"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "c136fcf1602e6bac485a92907782fe5d"
server_key_iv: "af114c7eb71078c5abca080e4a126411"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.439179 8636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:54.439838 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.440001 8636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:31:54.440099 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "20318447a752469eb0ba4e766821ab1e"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "c136fcf1602e6bac485a92907782fe5d"
server_key_iv: "af114c7eb71078c5abca080e4a126411"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.440173 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:54.455744 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:54.457988 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:54.458154 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:54.458406 8636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:54.458745 8636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:54.458797 8636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.458840 8636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:54.458871 8636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.464777 8636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36387
I20260620 14:31:54.464815 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36387 every 8 connection(s)
I20260620 14:31:54.465147 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:54.467094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8636
I20260620 14:31:54.467216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:54.467423 392 external_mini_cluster.cc:1428] Setting key eb1cd6db4a0441866270b8ba5da8d477
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:31:54.475487 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.465917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45287 (local address 127.0.98.62:42463)
0620 14:31:54.466042 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:54.466044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:54.466268 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:31:54.467399 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:54.467401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:54.467414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:54.467495 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:54.468104 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.468882 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.470064 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.470299 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.470387 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.472605 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:54.472607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:54.472607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:54.472616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:54.474091 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.474501 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.474502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.474503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.474559 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.474806 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.474806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.474807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.474921 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:54.474993 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:54.475305 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:31:54.475393 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:31:54.475685 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.465910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:45287)
0620 14:31:54.466141 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:54.466183 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:54.467284 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:54.467571 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:54.467576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:54.467840 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:54.468033 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.468037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.469050 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.469051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.469962 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.469965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.470397 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.470399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.470531 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.471131 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:54.471141 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:54.472498 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:54.474182 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.474189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.474191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.474415 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.474630 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.474631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.474631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.474736 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.474984 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:54.474991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:54.475169 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:31:54.475273 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:31:54.475434 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":123,"threads_started":1}
I20260620 14:31:54.476197 8765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260620 14:31:54.476300 8765 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:54.476485 8765 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
I20260620 14:31:54.476979 8574 ts_manager.cc:194] Registered new tserver with Master: 20318447a752469eb0ba4e766821ab1e (127.0.98.1:36387)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:31:54.477871 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:31:54.490288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:54.570914 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:54.571154 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:54.571178 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:54.572288 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:54.572330 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:54.572355 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:54.573874 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:54.574080 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:54.574271 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:54.579236 8770 server_base.cc:1034] running on GCE node
W20260620 14:31:54.579213 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.579195 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.579195 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.579646 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:54.579893 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:54.581077 8770 hybrid_clock.cc:648] HybridClock initialized: now 1781965914581058 us; error 30 us; skew 500 ppm
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:54.583617 8770 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:31:54.584046 8770 webserver.cc:460] Webserver started at http://127.0.98.2:42745/ using document root <none> and password file <none>
I20260620 14:31:54.584290 8770 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:54.584357 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:54.584504 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:54.585623 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab214a8f3df84b5f84a264c4fd0a4fc6"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "77fd019096079e26fccf47ee11549953"
server_key_iv: "6c85050afe7e25135254eed9358c3b3e"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.585978 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab214a8f3df84b5f84a264c4fd0a4fc6"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "77fd019096079e26fccf47ee11549953"
server_key_iv: "6c85050afe7e25135254eed9358c3b3e"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.587287 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:54.588116 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.588277 8770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:54.588371 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "ab214a8f3df84b5f84a264c4fd0a4fc6"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "77fd019096079e26fccf47ee11549953"
server_key_iv: "6c85050afe7e25135254eed9358c3b3e"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.588441 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:54.602742 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:54.604666 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:54.604831 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:54.605077 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:54.605388 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:54.605522 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.605572 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:54.605604 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.611408 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260620 14:31:54.611459 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260620 14:31:54.611788 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:54.614961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8770
I20260620 14:31:54.615098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:54.615315 392 external_mini_cluster.cc:1428] Setting key 5dd72bbabc2db40cd6e56dc43b7eb379
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:54.621896 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.612565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48437 (local address 127.0.98.62:42463)
0620 14:31:54.612626 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:54.612628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:54.612819 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:31:54.613798 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:54.613799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:54.613809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:54.613870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:54.614462 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.614969 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.616424 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.616608 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.616662 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.618953 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:54.618955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:54.618955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:54.618963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:54.620443 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.620920 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.620922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.620922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.620979 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.621227 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.621228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.621229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.621343 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:54.621416 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:54.621716 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:31:54.621802 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:31:54.622105 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.612536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:48437)
0620 14:31:54.612688 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:54.612731 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:54.613699 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:54.613932 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:54.613935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:54.614195 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:54.614387 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.614391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.615063 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.615064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.616340 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.616343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.616684 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.616685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.616830 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.617462 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:54.617473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:54.618854 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:54.620575 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.620582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.620585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.620828 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.621042 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.621043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.621043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.621161 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.621408 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:54.621414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:54.621586 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 14:31:54.621697 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:31:54.621862 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260620 14:31:54.622603 8899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260620 14:31:54.622720 8899 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:54.622928 8899 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:31:54.623346 8574 ts_manager.cc:194] Registered new tserver with Master: ab214a8f3df84b5f84a264c4fd0a4fc6 (127.0.98.2:43717)
I20260620 14:31:54.623823 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:31:54.639724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:54.727444 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:54.727767 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:54.727811 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:54.729697 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:54.729765 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:54.729835 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:54.732251 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:54.732596 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:54.732940 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:54.740818 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.740833 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.741154 8904 server_base.cc:1034] running on GCE node
W20260620 14:31:54.741248 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.741590 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:54.741927 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:54.743119 8904 hybrid_clock.cc:648] HybridClock initialized: now 1781965914743104 us; error 26 us; skew 500 ppm
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:54.745688 8904 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:31:54.746145 8904 webserver.cc:460] Webserver started at http://127.0.98.3:45973/ using document root <none> and password file <none>
I20260620 14:31:54.746376 8904 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:54.746438 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:54.746590 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:54.747641 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c5a5ebd9d7e45a39c98f3e9ea5221f0"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "73084b866ce8c4fef7a689412b433d4a"
server_key_iv: "42c94c804b62ff90f24d4887f7cdc705"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.747991 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c5a5ebd9d7e45a39c98f3e9ea5221f0"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "73084b866ce8c4fef7a689412b433d4a"
server_key_iv: "42c94c804b62ff90f24d4887f7cdc705"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.749270 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:31:54.750077 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.750264 8904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:54.750350 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "6c5a5ebd9d7e45a39c98f3e9ea5221f0"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "73084b866ce8c4fef7a689412b433d4a"
server_key_iv: "42c94c804b62ff90f24d4887f7cdc705"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.750432 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:54.764885 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:54.767455 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:54.767633 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:54.767879 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:54.768178 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:54.768229 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.768270 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:54.768296 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.773977 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46727
I20260620 14:31:54.774017 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46727 every 8 connection(s)
I20260620 14:31:54.774380 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:54.774852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 8904
I20260620 14:31:54.774945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:54.775117 392 external_mini_cluster.cc:1428] Setting key 592261ac46c2eed4dd8ca36b01691760
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:31:54.784312 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.775719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56053 (local address 127.0.98.62:42463)
0620 14:31:54.775782 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:54.775784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:54.776053 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:31:54.776980 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:54.776981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:54.776990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:54.777039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:54.777684 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.778172 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.779110 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.779297 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.779351 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.781518 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:54.781520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:54.781520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:54.781530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:54.783029 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.783414 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.783414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.783415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.783461 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:54.783662 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:54.783663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:54.783663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:54.783778 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:54.783852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:54.784137 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:31:54.784213 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:31:54.784515 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:54.775733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:56053)
0620 14:31:54.775922 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:54.775971 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:54.776896 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:54.777094 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:54.777097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:54.777351 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:54.777614 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.777619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.778243 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.778243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.779036 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:54.779037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:54.779373 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:54.779374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:54.779473 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:54.780015 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:54.780023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:54.781356 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:54.783108 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.783114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.783116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.783350 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.783513 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:54.783514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:54.783514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:54.783612 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:54.783847 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:54.783854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:54.783998 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:31:54.784114 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:31:54.784280 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260620 14:31:54.785017 9033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260620 14:31:54.785130 9033 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:54.785317 9033 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
I20260620 14:31:54.785697 8574 ts_manager.cc:194] Registered new tserver with Master: 6c5a5ebd9d7e45a39c98f3e9ea5221f0 (127.0.98.3:46727)
I20260620 14:31:54.786118 8574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56053
I20260620 14:31:54.786895 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:31:54.855875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8636
I20260620 14:31:54.860379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8770
I20260620 14:31:54.864635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8904
I20260620 14:31:54.868925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 8542
I20260620 14:31:54.873931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--webserver_interface=127.0.98.62
--webserver_port=39849
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:54.954566 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:54.954777 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:54.954800 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:54.955969 9042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:54.956012 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:54.956027 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:54.956039 9042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:54.956053 9042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:54.957541 9042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:54.957746 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:54.957930 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:54.962658 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.962666 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:54.962675 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:54.962858 9042 server_base.cc:1034] running on GCE node
I20260620 14:31:54.963111 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:54.963397 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:54.964555 9042 hybrid_clock.cc:648] HybridClock initialized: now 1781965914964559 us; error 45 us; skew 500 ppm
Jun 20 14:31:54 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:54.967067 9042 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:31:54.967518 9042 webserver.cc:460] Webserver started at http://127.0.98.62:39849/ using document root <none> and password file <none>
I20260620 14:31:54.967764 9042 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:54.967844 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:54.969137 9042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:54.969818 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.969970 9042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:54.970065 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "43c4a3262e6242979f77a9bad7963f65"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "a763ce5bf169dee5f6b7bab666c8b115"
server_key_iv: "4d045ba74fac4aea682125301cfdbbac"
server_key_version: "encryptionkey@0"
I20260620 14:31:54.970396 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:54.979442 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:54.981797 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:54.981957 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:54.985749 9042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42463
I20260620 14:31:54.985785 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42463 every 8 connection(s)
I20260620 14:31:54.986330 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:54.987525 9110 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:31:54.988224 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Bootstrap starting.
I20260620 14:31:54.988411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9042
I20260620 14:31:54.988689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36387
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42463
--webserver_port=34173
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:31:54.996358 9110 log.cc:824] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:54.998062 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:31:54.998359 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Bootstrap complete.
I20260620 14:31:55.000557 9110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:55.000800 9110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c4a3262e6242979f77a9bad7963f65, State: Initialized, Role: FOLLOWER
I20260620 14:31:55.000969 9110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:55.001040 9110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:55.001089 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:55.001137 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:55.002162 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:55.002302 9110 leader_election.cc:302] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c4a3262e6242979f77a9bad7963f65; no voters:
I20260620 14:31:55.002492 9110 leader_election.cc:288] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:31:55.002527 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:31:55.002637 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [term 2 LEADER]: Becoming Leader. State: Replica: 43c4a3262e6242979f77a9bad7963f65, State: Running, Role: LEADER
I20260620 14:31:55.002723 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260620 14:31:55.002767 9110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:55.002972 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c4a3262e6242979f77a9bad7963f65. Latest consensus state: current_term: 2 leader_uuid: "43c4a3262e6242979f77a9bad7963f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260620 14:31:55.003182 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:55.003126 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43c4a3262e6242979f77a9bad7963f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c4a3262e6242979f77a9bad7963f65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260620 14:31:55.003405 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:55.003533 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:55.004031 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:55.004166 9122 catalog_manager.cc:1259] Loaded cluster ID: 96dea574aa7742f7ab402158bf9091ec
I20260620 14:31:55.004369 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:55.005539 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:55.005717 9122 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43c4a3262e6242979f77a9bad7963f65: Loaded TSK: 0
I20260620 14:31:55.005939 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:31:55.071087 9112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:55.071324 9112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:55.071367 9112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:55.072599 9112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:55.072642 9112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:55.072669 9112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:31:55.074249 9112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34173
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:55.074465 9112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:55.074697 9112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:55.079779 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:55.079772 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.080015 9112 server_base.cc:1034] running on GCE node
W20260620 14:31:55.079767 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.080317 9112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:55.080586 9112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:55.081758 9112 hybrid_clock.cc:648] HybridClock initialized: now 1781965915081738 us; error 35 us; skew 500 ppm
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965915, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:55.084807 9112 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:31:55.085389 9112 webserver.cc:460] Webserver started at http://127.0.98.1:34173/ using document root <none> and password file <none>
I20260620 14:31:55.085678 9112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:55.085762 9112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:55.087579 9112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:55.088225 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.088423 9112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:55.088521 9112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "20318447a752469eb0ba4e766821ab1e"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "c136fcf1602e6bac485a92907782fe5d"
server_key_iv: "af114c7eb71078c5abca080e4a126411"
server_key_version: "encryptionkey@0"
I20260620 14:31:55.088972 9112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:55.116076 9112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:55.118571 9112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:55.118731 9112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:55.119012 9112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:55.119382 9112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:55.119433 9112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.119467 9112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:55.119490 9112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.126724 9112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36387
I20260620 14:31:55.126762 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36387 every 8 connection(s)
I20260620 14:31:55.127167 9112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:55.133745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9112
I20260620 14:31:55.134145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43717
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42463
--webserver_port=42745
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.135855 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.127950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:48353)
0620 14:31:55.128127 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.128175 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.128677 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.129188 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.129191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.129843 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.130043 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.130048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.131165 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.131166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.132012 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.132015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.132626 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.132627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.132753 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.133295 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.133303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.135633 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
W20260620 14:31:55.136090 9260 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42463: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260620 14:31:55.136622 9263 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.127998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48353 (local address 127.0.98.62:42463)
0620 14:31:55.128204 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.128211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.128273 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:31:55.128898 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.128903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.128975 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.129147 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.130170 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.131055 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.132104 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.132525 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.132653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.136420 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48353: BlockingRecv error: recv got EOF from 127.0.98.1:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.142405 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.136300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:57445)
0620 14:31:55.136617 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.136628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.136668 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.136994 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.136995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.137151 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.137204 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.137206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.138224 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.138226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.139141 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.139143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.139679 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.139680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.139750 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.140233 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.140238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.142309 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":81,"threads_started":1}
W20260620 14:31:55.142656 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.136469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57445 (local address 127.0.98.62:42463)
0620 14:31:55.136788 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.136790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.136796 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:31:55.136817 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.136818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.136847 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.137270 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.137303 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.138121 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.139243 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.139586 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.141279 (+ 1693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.142416 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57445: BlockingRecv error: recv got EOF from 127.0.98.1:57445 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":79,"threads_started":1}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.147641 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.142687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:58265)
0620 14:31:55.142847 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.142858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.142904 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.143107 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.143108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.143310 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.143350 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.143352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.144127 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.144128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.145077 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.145080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.145408 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.145409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.145505 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.145900 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.145905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.147587 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260620 14:31:55.147785 9260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:31:55.147907 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.142778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58265 (local address 127.0.98.62:42463)
0620 14:31:55.142836 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.142839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.142905 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:31:55.142971 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.142972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.142982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.143044 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.143445 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.144027 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.145160 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.145341 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.145386 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.147684 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58265: BlockingRecv error: recv got EOF from 127.0.98.1:58265 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:31:55.219035 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:55.219254 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:55.219277 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:55.220364 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:55.220397 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:55.220422 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:31:55.221814 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42745
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:55.222013 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:55.222196 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:55.227031 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:55.227092 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.227252 9264 server_base.cc:1034] running on GCE node
W20260620 14:31:55.227125 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.227521 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:55.227766 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:55.228904 9264 hybrid_clock.cc:648] HybridClock initialized: now 1781965915228886 us; error 36 us; skew 500 ppm
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965915, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:55.231370 9264 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:31:55.231832 9264 webserver.cc:460] Webserver started at http://127.0.98.2:42745/ using document root <none> and password file <none>
I20260620 14:31:55.232050 9264 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:55.232113 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:55.233597 9264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:31:55.234167 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.234328 9264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:55.234426 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "ab214a8f3df84b5f84a264c4fd0a4fc6"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "77fd019096079e26fccf47ee11549953"
server_key_iv: "6c85050afe7e25135254eed9358c3b3e"
server_key_version: "encryptionkey@0"
I20260620 14:31:55.234766 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:55.265383 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:55.267289 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:55.267460 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:55.267721 9264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:55.268030 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:55.268088 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.268132 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:55.268159 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.274520 9264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260620 14:31:55.274560 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260620 14:31:55.274906 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:55.280035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9264
I20260620 14:31:55.280392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46727
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42463
--webserver_port=45973
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.282267 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.275593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:43479)
0620 14:31:55.275732 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.275766 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.276227 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.276495 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.276498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.277054 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.277244 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.277249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.277963 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.277964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.278798 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.278801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.279117 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.279118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.279213 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.279681 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.279690 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.282121 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260620 14:31:55.282325 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.275596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43479 (local address 127.0.98.62:42463)
0620 14:31:55.275655 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.275657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.275841 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:31:55.276334 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.276335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.276350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.276408 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.277323 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.277883 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.278871 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.279053 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.279103 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.282243 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43479: BlockingRecv error: recv got EOF from 127.0.98.2:43479 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:31:55.282554 9395 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42463: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.287184 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.282784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:52851)
0620 14:31:55.282829 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.282838 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.282890 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.283029 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.283029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.283172 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.283213 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.283215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.283848 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.283849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.284356 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.284357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.284716 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.284717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.284777 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.285253 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.285258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.287102 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:31:55.287236 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.282836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52851 (local address 127.0.98.62:42463)
0620 14:31:55.282894 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.282896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.282904 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:55.282947 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.282948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.282962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.283027 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.283288 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.283759 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.284447 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.284639 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.284717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.287172 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52851: BlockingRecv error: recv got EOF from 127.0.98.2:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.291560 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.287422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:58183)
0620 14:31:55.287563 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.287571 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.287629 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.287765 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.287766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.287882 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.287906 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.287908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.288504 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.288505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.288994 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.288995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.289331 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.289332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.289383 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.289879 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.289883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.291478 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260620 14:31:55.291616 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.287466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58183 (local address 127.0.98.62:42463)
0620 14:31:55.287503 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.287504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.287616 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:31:55.287690 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.287691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.287699 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.287762 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.287975 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.288423 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.289071 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.289253 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.289331 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.291546 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58183: BlockingRecv error: recv got EOF from 127.0.98.2:58183 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:31:55.291685 9395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:31:55.362551 9398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:55.362792 9398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:55.362831 9398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:55.364003 9398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:55.364073 9398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:55.364126 9398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:31:55.365680 9398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45973
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:31:55.365943 9398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:55.366173 9398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:55.371307 9405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:55.371292 9404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.371349 9398 server_base.cc:1034] running on GCE node
W20260620 14:31:55.371291 9407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:55.371698 9398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:55.371982 9398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:55.373142 9398 hybrid_clock.cc:648] HybridClock initialized: now 1781965915373125 us; error 36 us; skew 500 ppm
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965915, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:55.375566 9398 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:31:55.375983 9398 webserver.cc:460] Webserver started at http://127.0.98.3:45973/ using document root <none> and password file <none>
I20260620 14:31:55.376205 9398 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:55.376286 9398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:55.377631 9398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:55.378222 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.378372 9398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:31:55.378466 9398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "6c5a5ebd9d7e45a39c98f3e9ea5221f0"
format_stamp: "Formatted at 2026-06-20 14:31:54 on dist-test-slave-tc2g"
server_key: "73084b866ce8c4fef7a689412b433d4a"
server_key_iv: "42c94c804b62ff90f24d4887f7cdc705"
server_key_version: "encryptionkey@0"
I20260620 14:31:55.378801 9398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:55.394129 9398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:55.396169 9398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:55.396327 9398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:55.396579 9398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:55.396876 9398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:55.396927 9398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.396968 9398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:55.396999 9398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:55.403148 9398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46727
I20260620 14:31:55.403223 9527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46727 every 8 connection(s)
I20260620 14:31:55.403604 9398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:55.405654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9398
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:55.412106 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.404353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51001 (local address 127.0.98.62:42463)
0620 14:31:55.404422 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.404424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.404672 (+ 248us) server_negotiation.cc:407] Connection header received
0620 14:31:55.405155 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.405156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.405168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.405223 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.406345 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.406914 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.408214 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.408388 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.408461 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.412056 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51001: BlockingRecv error: recv got EOF from 127.0.98.3:51001 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:31:55.412225 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.404367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:51001)
0620 14:31:55.404546 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.404584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.405068 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.405279 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.405282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.405999 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.406250 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.406258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.407002 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.407003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.408107 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.408109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.408469 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.408470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.408588 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.409115 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.409125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.411986 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260620 14:31:55.412451 9528 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42463: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 14:31:55.413635 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.406168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.0.98.62:42463)
0620 14:31:55.406207 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.406209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.406316 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:31:55.406387 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.406387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.406400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.406462 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.407054 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.407495 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.408467 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.408671 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.408714 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.410060 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:55.410066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:55.410066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:55.410078 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:55.412151 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:55.412557 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:55.412561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:55.412561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:55.412658 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:55.412891 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:55.412893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:55.412893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:55.413163 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:55.413303 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:55.413400 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:31:55.413531 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.416888 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.412722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:51829)
0620 14:31:55.412769 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.412779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.412816 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.412956 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.412957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.413069 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.413113 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.413114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.413933 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.413934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.414658 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.414660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.415008 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.415008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.415072 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.415398 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.415402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.416845 (+ 1443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:31:55.416986 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.412754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51829 (local address 127.0.98.62:42463)
0620 14:31:55.412807 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.412808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.412815 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:31:55.412880 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.412881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.412890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.412944 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.413180 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.413853 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.414737 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.414933 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.415008 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.416906 (+ 1898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51829: BlockingRecv error: recv got EOF from 127.0.98.3:51829 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:55 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:55.420581 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:55.417128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:59999)
0620 14:31:55.417162 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:55.417171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:55.417203 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:55.417302 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:55.417302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:55.417399 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:55.417470 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.417471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.418017 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.418017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.418475 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:55.418476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.418762 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.418762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:55.418796 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.419107 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:55.419111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:55.420537 (+ 1426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:31:55.420696 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:55.417168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59999 (local address 127.0.98.62:42463)
0620 14:31:55.417212 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:55.417214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:55.417222 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:55.417242 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:55.417243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:55.417255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:55.417301 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:55.417530 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.417953 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.418535 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:55.418706 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:55.418753 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:55.420626 (+ 1873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59999: BlockingRecv error: recv got EOF from 127.0.98.3:59999 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 14:31:55.420707 9528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:56.157728 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:56.148268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:58657)
0620 14:31:56.148409 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:56.148424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:56.148472 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:56.148629 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:56.148630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:56.148787 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:56.148875 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.148877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.149577 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.149578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.150109 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.150111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.150435 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.150435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.150480 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.155838 (+ 5358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:56.155843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:56.157663 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
W20260620 14:31:56.157856 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:56.148382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58657 (local address 127.0.98.62:42463)
0620 14:31:56.148512 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:56.148513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:56.148521 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:31:56.148542 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:56.148542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:56.148552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:56.148616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:56.148965 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.149514 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.150197 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.150378 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.150428 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.157771 (+ 7343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58657: BlockingRecv error: recv got EOF from 127.0.98.1:58657 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:56.297139 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:56.292151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:59021)
0620 14:31:56.292277 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:56.292288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:56.292348 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:56.292516 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:56.292517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:56.292704 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:56.292752 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.292753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.293373 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.293373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.294017 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.294019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.294434 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.294435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.294474 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.294905 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:56.294909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:56.297065 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260620 14:31:56.297248 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:56.292211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59021 (local address 127.0.98.62:42463)
0620 14:31:56.292272 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:56.292277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:56.292334 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:31:56.292391 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:56.292392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:56.292402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:56.292500 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:56.292810 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.293311 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.294172 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.294364 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.294428 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.297157 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59021: BlockingRecv error: recv got EOF from 127.0.98.2:59021 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:56 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:56.425591 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:56.421138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:37573)
0620 14:31:56.421272 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:56.421282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:56.421332 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:56.421489 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:56.421489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:56.421655 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:56.421702 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.421704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.422314 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.422315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.422866 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:56.422867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.423162 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.423162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:56.423198 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.423620 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:56.423624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:56.425491 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260620 14:31:56.425621 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:56.421248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37573 (local address 127.0.98.62:42463)
0620 14:31:56.421308 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:56.421310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:56.421331 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:31:56.421395 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:56.421395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:56.421406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:56.421471 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:56.421772 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.422248 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.422935 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:56.423107 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:56.423154 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:56.425550 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37573: BlockingRecv error: recv got EOF from 127.0.98.3:37573 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:57.162806 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:57.158203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:32895)
0620 14:31:57.158448 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:57.158458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:57.158516 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:57.158673 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:57.158673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:57.158849 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:57.158894 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.158896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.159514 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.159514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.160034 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.160035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.160345 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.160346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.160382 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.160894 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:57.160899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:57.162748 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":189,"threads_started":1}
W20260620 14:31:57.162947 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:57.158266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32895 (local address 127.0.98.62:42463)
0620 14:31:57.158393 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:57.158395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:57.158514 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:31:57.158583 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:57.158583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:57.158593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:57.158658 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:57.158954 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.159446 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.160104 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.160284 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.160337 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.162855 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32895: BlockingRecv error: recv got EOF from 127.0.98.1:32895 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:57.302191 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:57.297589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:38505)
0620 14:31:57.297757 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:57.297770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:57.297849 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:57.297995 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:57.297996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:57.298197 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:57.298246 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.298249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.298940 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.298941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.299471 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.299472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.299836 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.299837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.299899 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.300334 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:57.300338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:57.302129 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
W20260620 14:31:57.302322 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:57.297669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38505 (local address 127.0.98.62:42463)
0620 14:31:57.297742 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:57.297745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:57.297832 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:31:57.297931 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:57.297932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:57.297944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:57.297992 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:57.298326 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.298850 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.299558 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.299751 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.299835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.302239 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38505: BlockingRecv error: recv got EOF from 127.0.98.2:38505 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:57 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:57.430306 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:57.426124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:44995)
0620 14:31:57.426283 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:57.426294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:57.426352 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:57.426465 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:57.426466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:57.426613 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:57.426663 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.426665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.427276 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.427277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.427778 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:57.427779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.428069 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.428070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:57.428106 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.428522 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:57.428526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:57.430243 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
W20260620 14:31:57.430438 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:57.426234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44995 (local address 127.0.98.62:42463)
0620 14:31:57.426282 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:57.426284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:57.426339 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:31:57.426404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:57.426404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:57.426418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:57.426453 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:57.426720 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.427211 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.427843 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:57.428014 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:57.428061 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:57.430354 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44995: BlockingRecv error: recv got EOF from 127.0.98.3:44995 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:58.167841 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:58.163356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:43113)
0620 14:31:58.163521 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:58.163533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:58.163594 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:58.163728 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:58.163728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:58.163884 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:58.163934 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.163935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.164554 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.164555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.165062 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.165064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.165363 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.165364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.165400 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.165933 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:58.165939 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:58.167783 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
W20260620 14:31:58.167979 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:58.163465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43113 (local address 127.0.98.62:42463)
0620 14:31:58.163581 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:58.163583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:58.163599 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:31:58.163649 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:58.163650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:58.163659 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:58.163723 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:58.163990 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.164485 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.165127 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.165305 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.165353 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.167887 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43113: BlockingRecv error: recv got EOF from 127.0.98.1:43113 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:58.307278 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:58.302632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:54047)
0620 14:31:58.302831 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:58.302840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:58.302922 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:58.303073 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:58.303074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:58.303243 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:58.303291 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.303293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.303936 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.303937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.304529 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.304531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.304843 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.304844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.304883 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.305334 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:58.305339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:58.307220 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
W20260620 14:31:58.307390 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:58.302711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54047 (local address 127.0.98.62:42463)
0620 14:31:58.302769 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:58.302771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:58.302918 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:31:58.302980 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:58.302980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:58.302991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:58.303059 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:58.303354 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.303862 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.304597 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.304781 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.304833 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.307300 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54047: BlockingRecv error: recv got EOF from 127.0.98.2:54047 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:58 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:58.435249 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:58.430785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:55257)
0620 14:31:58.430934 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:58.430945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:58.431018 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:58.431147 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:58.431148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:58.431303 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:58.431351 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.431353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.431963 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.431964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.432478 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:58.432480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.432787 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.432788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:58.432827 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.433332 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:58.433337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:58.435138 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
W20260620 14:31:58.435283 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:58.430886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55257 (local address 127.0.98.62:42463)
0620 14:31:58.430957 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:58.430960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:58.431012 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:31:58.431065 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:58.431065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:58.431078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:58.431128 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:58.431407 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.431900 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.432542 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:58.432729 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:58.432777 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:58.435223 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55257: BlockingRecv error: recv got EOF from 127.0.98.3:55257 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:59.174045 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:59.168346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:40421)
0620 14:31:59.168479 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:59.168490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:59.168546 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:59.168721 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:59.168722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:59.168873 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:59.168919 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.168920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.170032 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.170033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.170576 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.170578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.171136 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.171136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.171198 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.171764 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:59.171769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:59.173928 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260620 14:31:59.174060 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:59.168396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40421 (local address 127.0.98.62:42463)
0620 14:31:59.168522 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:59.168524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:59.168543 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:31:59.168622 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:59.168622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:59.168637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:59.168710 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:59.168988 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.169931 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.170663 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.171050 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.171129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.174007 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40421: BlockingRecv error: recv got EOF from 127.0.98.1:40421 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":50,"threads_started":1}
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:59.314555 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:59.307766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:55569)
0620 14:31:59.307924 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:59.307938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:59.308013 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:59.308220 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:59.308221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:59.308459 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:59.308541 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.308543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.309449 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.309450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.310878 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.310881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.311404 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.311406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.311470 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.312011 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:59.312016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:59.314445 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
W20260620 14:31:59.314692 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:59.307860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55569 (local address 127.0.98.62:42463)
0620 14:31:59.307914 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:59.307916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:59.308020 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:31:59.308089 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:59.308090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:59.308100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:59.308167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:59.308623 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.309325 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.310986 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.311321 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.311403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.314555 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55569: BlockingRecv error: recv got EOF from 127.0.98.2:55569 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:59 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:59.440615 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:59.435783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:49369)
0620 14:31:59.435949 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:59.435959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:59.436046 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:59.436224 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:59.436225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:59.436399 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:59.436445 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.436447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.437160 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.437161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.437779 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:59.437782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.438133 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.438134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:59.438192 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.438716 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:59.438720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:59.440517 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
W20260620 14:31:59.440663 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:59.435842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49369 (local address 127.0.98.62:42463)
0620 14:31:59.435907 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:59.435909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:59.436026 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:31:59.436118 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:59.436119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:59.436133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:59.436197 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:59.436523 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.437065 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.437876 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:59.438064 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:59.438134 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:59.440597 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49369: BlockingRecv error: recv got EOF from 127.0.98.3:49369 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:00.178985 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:00.174499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:44683)
0620 14:32:00.174641 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:00.174654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:00.174725 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:00.174860 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:00.174861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:00.175022 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:00.175071 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.175073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.175707 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.175707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.176226 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.176228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.176529 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.176530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.176566 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.177006 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:00.177010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:00.178888 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
W20260620 14:32:00.179047 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:00.174542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44683 (local address 127.0.98.62:42463)
0620 14:32:00.174679 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:00.174680 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:00.174722 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:32:00.174776 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:00.174776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:00.174786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:00.174845 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:00.175139 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.175638 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.176290 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.176472 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.176521 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.178973 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44683: BlockingRecv error: recv got EOF from 127.0.98.1:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:00.319341 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:00.314997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:36055)
0620 14:32:00.315159 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:00.315177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:00.315237 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:00.315360 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:00.315360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:00.315522 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:00.315571 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.315572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.316186 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.316187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.316708 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.316709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.317042 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.317042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.317082 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.317595 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:00.317601 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:00.319269 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
W20260620 14:32:00.319410 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:00.315180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36055 (local address 127.0.98.62:42463)
0620 14:32:00.315242 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:00.315244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:00.315258 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:32:00.315282 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:00.315283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:00.315292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:00.315343 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:00.315632 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.316121 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.316797 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.316980 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.317032 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.319364 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36055: BlockingRecv error: recv got EOF from 127.0.98.2:36055 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:00 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:00.446204 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:00.441259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:45919)
0620 14:32:00.441386 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:00.441403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:00.441507 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:00.441652 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:00.441652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:00.441814 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:00.441861 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.441862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.442811 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.442812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.443358 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:00.443360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.443695 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.443696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:00.443738 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.444192 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:00.444197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:00.446085 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
W20260620 14:32:00.446247 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:00.441351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45919 (local address 127.0.98.62:42463)
0620 14:32:00.441411 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:00.441413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:00.441490 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:32:00.441557 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:00.441558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:00.441570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:00.441644 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:00.441940 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.442741 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.443433 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:00.443633 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:00.443690 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:00.446175 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45919: BlockingRecv error: recv got EOF from 127.0.98.3:45919 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:01.188721 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:01.179448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:43135)
0620 14:32:01.179618 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:01.179631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:01.179717 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:01.179955 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:01.179956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:01.180116 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:01.180163 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.180165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.180957 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.180958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.181566 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.181569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.181983 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.181985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.182048 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.186801 (+ 4753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:01.186807 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:01.188606 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
W20260620 14:32:01.188781 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:01.179516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43135 (local address 127.0.98.62:42463)
0620 14:32:01.179620 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:01.179622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:01.179696 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:32:01.179791 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:01.179792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:01.179811 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:01.179899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:01.180260 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.180747 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.181674 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.181909 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.181982 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.188704 (+ 6722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43135: BlockingRecv error: recv got EOF from 127.0.98.1:43135 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:01.324663 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:01.319938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:55575)
0620 14:32:01.320068 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:01.320081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:01.320157 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:01.320401 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:01.320401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:01.320606 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:01.320663 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.320664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.321371 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.321372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.321952 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.321954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.322310 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.322311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.322368 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.322804 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:01.322808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:01.324555 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260620 14:32:01.324704 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:01.320069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55575 (local address 127.0.98.62:42463)
0620 14:32:01.320133 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:01.320135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:01.320148 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:32:01.320240 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:01.320242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:01.320256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:01.320332 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:01.320742 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.321278 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.322040 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.322231 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.322310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.324640 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55575: BlockingRecv error: recv got EOF from 127.0.98.2:55575 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:01 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:01.451637 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:01.446724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:58021)
0620 14:32:01.446902 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:01.446913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:01.447000 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:01.447156 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:01.447157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:01.447351 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:01.447397 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.447399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.448065 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.448066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.448598 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:01.448600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.448959 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.448960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:01.449017 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.449569 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:01.449575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:01.451539 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
W20260620 14:32:01.451683 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:01.446814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58021 (local address 127.0.98.62:42463)
0620 14:32:01.446881 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:01.446883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:01.446979 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:32:01.447066 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:01.447067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:01.447081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:01.447151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:01.447476 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.447984 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.448685 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:01.448878 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:01.448957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:01.451617 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58021: BlockingRecv error: recv got EOF from 127.0.98.3:58021 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:02.194176 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:02.189358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:38763)
0620 14:32:02.189558 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:02.189571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:02.189638 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:02.189796 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:02.189797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:02.189957 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:02.190005 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.190006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.190824 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.190825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.191399 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.191401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.191733 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.191734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.191785 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.192218 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:02.192222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:02.194052 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
W20260620 14:32:02.194206 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:02.189503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38763 (local address 127.0.98.62:42463)
0620 14:32:02.189629 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:02.189631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:02.189641 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:32:02.189689 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:02.189689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:02.189699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:02.189766 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:02.190068 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.190691 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.191476 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.191667 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.191724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.194134 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38763: BlockingRecv error: recv got EOF from 127.0.98.1:38763 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:02.330077 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:02.325173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:59409)
0620 14:32:02.325486 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:02.325497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:02.325566 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:02.325719 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:02.325720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:02.325925 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:02.325972 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.325974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.326663 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.326664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.327199 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.327200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.327590 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.327591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.327649 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.328117 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:02.328122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:02.329979 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
W20260620 14:32:02.330137 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:02.325371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59409 (local address 127.0.98.62:42463)
0620 14:32:02.325469 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:02.325471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:02.325547 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:32:02.325621 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:02.325622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:02.325636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:02.325711 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:02.326059 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.326566 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.327303 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.327502 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.327589 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.330073 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59409: BlockingRecv error: recv got EOF from 127.0.98.2:59409 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:02 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:02.457048 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:02.452285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:59269)
0620 14:32:02.452506 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:02.452523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:02.452620 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:02.452852 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:02.452853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:02.453074 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:02.453136 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.453139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.453877 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.453878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.454404 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:02.454405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.454708 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.454709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:02.454750 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.455210 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:02.455214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:02.456989 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
W20260620 14:32:02.457155 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:02.452400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59269 (local address 127.0.98.62:42463)
0620 14:32:02.452479 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:02.452482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:02.452621 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:32:02.452682 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:02.452684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:02.452698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:02.452779 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:02.453225 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.453807 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.454475 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:02.454647 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:02.454695 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:02.457084 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59269: BlockingRecv error: recv got EOF from 127.0.98.3:59269 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:03.199139 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:03.194620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:46299)
0620 14:32:03.194757 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:03.194767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:03.194822 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:03.195003 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:03.195004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:03.195165 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:03.195211 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.195212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.195872 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.195873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.196418 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.196419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.196738 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.196739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.196776 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.197210 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:03.197214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:03.199080 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
W20260620 14:32:03.199344 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:03.194761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46299 (local address 127.0.98.62:42463)
0620 14:32:03.194887 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:03.194889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:03.194897 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:32:03.194918 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:03.194919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:03.194931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:03.194990 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:03.195309 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.195801 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.196487 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.196678 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.196727 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.199267 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46299: BlockingRecv error: recv got EOF from 127.0.98.1:46299 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:03.335544 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:03.330684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:44649)
0620 14:32:03.330874 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:03.330889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:03.330970 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:03.331113 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:03.331114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:03.331289 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:03.331339 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.331340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.332178 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.332179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.332968 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.332970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.333290 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.333291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.333328 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.333816 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:03.333821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:03.335485 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
W20260620 14:32:03.335675 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:03.330855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44649 (local address 127.0.98.62:42463)
0620 14:32:03.330907 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:03.330909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:03.330953 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:32:03.331028 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:03.331029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:03.331039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:03.331089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:03.331403 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.332098 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.333021 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.333215 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.333270 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.335584 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44649: BlockingRecv error: recv got EOF from 127.0.98.2:44649 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:03 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:03.462110 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:03.457629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:36723)
0620 14:32:03.457765 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:03.457777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:03.457851 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:03.457979 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:03.457980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:03.458139 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:03.458188 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.458189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.458847 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.458848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.459394 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:03.459396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.459757 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.459758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:03.459801 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.460264 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:03.460269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:03.462049 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260620 14:32:03.462222 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:03.457683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36723 (local address 127.0.98.62:42463)
0620 14:32:03.457749 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:03.457751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:03.457851 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:32:03.457890 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:03.457891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:03.457902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:03.457959 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:03.458249 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.458725 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.459487 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:03.459672 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:03.459728 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:03.462150 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36723: BlockingRecv error: recv got EOF from 127.0.98.3:36723 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:04.204378 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:04.199652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:56471)
0620 14:32:04.199799 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:04.199812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:04.199877 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:04.200001 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:04.200002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:04.200159 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:04.200208 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.200210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.200822 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.200823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.201342 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.201344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.201711 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.201712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.201749 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.202246 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:04.202252 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:04.204273 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
W20260620 14:32:04.204428 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:04.199691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56471 (local address 127.0.98.62:42463)
0620 14:32:04.199812 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:04.199816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:04.199859 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:32:04.199927 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:04.199928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:04.199937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:04.199990 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:04.200264 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.200754 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.201410 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.201647 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.201701 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.204357 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56471: BlockingRecv error: recv got EOF from 127.0.98.1:56471 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:04.340955 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:04.336085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:34169)
0620 14:32:04.336228 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:04.336242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:04.336307 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:04.336452 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:04.336453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:04.336613 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:04.336663 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.336664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.337469 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.337470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.338068 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.338070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.338430 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.338431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.338479 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.338938 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:04.338943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:04.340854 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260620 14:32:04.341090 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:04.336168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34169 (local address 127.0.98.62:42463)
0620 14:32:04.336232 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:04.336234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:04.336311 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:32:04.336365 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:04.336366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:04.336377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:04.336432 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:04.336724 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.337348 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.338144 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.338345 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.338401 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.341027 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34169: BlockingRecv error: recv got EOF from 127.0.98.2:34169 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:04 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:04.467754 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:04.462801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:45669)
0620 14:32:04.462986 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:04.462998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:04.463076 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:04.463247 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:04.463249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:04.463434 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:04.463479 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.463481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.464157 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.464158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.464696 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:04.464697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.465078 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.465079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:04.465139 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.465645 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:04.465650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:04.467653 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
W20260620 14:32:04.467808 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:04.462863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45669 (local address 127.0.98.62:42463)
0620 14:32:04.462932 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:04.462935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:04.463061 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:32:04.463132 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:04.463133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:04.463154 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:04.463224 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:04.463561 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.464057 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.464788 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:04.464986 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:04.465075 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:04.467743 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45669: BlockingRecv error: recv got EOF from 127.0.98.3:45669 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:05.209842 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:05.204869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:53125)
0620 14:32:05.205022 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:05.205033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:05.205097 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:05.205252 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:05.205253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:05.205434 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:05.205479 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.205481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.206204 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.206205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.206734 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.206735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.207131 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.207133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.207185 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.207782 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:05.207791 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:05.209752 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260620 14:32:05.209924 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:05.204940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53125 (local address 127.0.98.62:42463)
0620 14:32:05.205059 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:05.205061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:05.205088 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:32:05.205162 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:05.205163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:05.205174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:05.205238 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:05.205541 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.206094 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.206825 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.207024 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.207102 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.209854 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53125: BlockingRecv error: recv got EOF from 127.0.98.1:53125 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":48,"threads_started":1}
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:05.345827 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:05.341573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:37121)
0620 14:32:05.341722 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:05.341731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:05.341788 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:05.341907 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:05.341907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:05.342063 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:05.342112 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.342114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.342717 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.342718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.343239 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.343241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.343554 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.343555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.343592 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.344015 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:05.344019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:05.345770 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
W20260620 14:32:05.345958 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:05.341634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37121 (local address 127.0.98.62:42463)
0620 14:32:05.341700 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:05.341701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:05.341790 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:32:05.341834 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:05.341834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:05.341846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:05.341897 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:05.342168 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.342651 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.343311 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.343494 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.343551 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.345880 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37121: BlockingRecv error: recv got EOF from 127.0.98.2:37121 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:05 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:05.473596 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:05.468399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:42019)
0620 14:32:05.468582 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:05.468596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:05.468683 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:05.468835 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:05.468836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:05.468995 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:05.469042 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.469044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.470082 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.470083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.470602 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:05.470604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.470941 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.470942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:05.470987 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.471450 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:05.471454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:05.473492 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
W20260620 14:32:05.473650 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:05.468623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42019 (local address 127.0.98.62:42463)
0620 14:32:05.468689 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:05.468691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:05.468729 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:32:05.468758 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:05.468759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:05.468770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:05.468811 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:05.469144 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.470004 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.470682 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:05.470869 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:05.470925 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:05.473568 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42019: BlockingRecv error: recv got EOF from 127.0.98.3:42019 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:06.219744 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:06.210512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:58057)
0620 14:32:06.210694 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:06.210706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:06.210778 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:06.210922 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:06.210923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:06.211088 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:06.211136 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.211138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.211826 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.211827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.212381 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.212384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.212790 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.212791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.212855 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.217766 (+ 4911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:06.217772 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:06.219651 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
W20260620 14:32:06.219825 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:06.210635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58057 (local address 127.0.98.62:42463)
0620 14:32:06.210787 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:06.210788 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:06.210799 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:32:06.210827 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:06.210828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:06.210838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:06.210903 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:06.211210 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.211728 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.212491 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.212696 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.212751 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.219751 (+ 7000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58057: BlockingRecv error: recv got EOF from 127.0.98.1:58057 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:06.350936 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:06.346257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:33377)
0620 14:32:06.346435 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:06.346455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:06.346529 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:06.346678 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:06.346679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:06.346840 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:06.346892 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.346894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.347550 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.347551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.348105 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.348107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.348429 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.348430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.348469 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.348925 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:06.348930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:06.350816 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
W20260620 14:32:06.350976 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:06.346381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33377 (local address 127.0.98.62:42463)
0620 14:32:06.346457 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:06.346459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:06.346523 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:32:06.346598 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:06.346598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:06.346608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:06.346651 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:06.346968 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.347466 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.348179 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.348356 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.348415 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.350911 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33377: BlockingRecv error: recv got EOF from 127.0.98.2:33377 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:06 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:06.478793 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:06.474200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:60041)
0620 14:32:06.474332 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:06.474342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:06.474410 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:06.474558 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:06.474559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:06.474719 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:06.474767 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.474768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.475401 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.475402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.475927 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:06.475928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.476246 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.476247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:06.476285 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.476733 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:06.476737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:06.478734 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
W20260620 14:32:06.478900 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:06.474224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60041 (local address 127.0.98.62:42463)
0620 14:32:06.474303 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:06.474305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:06.474416 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:32:06.474474 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:06.474476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:06.474490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:06.474536 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:06.474834 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.475326 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.476004 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:06.476181 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:06.476232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:06.478825 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60041: BlockingRecv error: recv got EOF from 127.0.98.3:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:07.225152 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:07.220383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:60885)
0620 14:32:07.220517 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:07.220528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:07.220579 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:07.220776 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:07.220777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:07.220948 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:07.220996 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.220997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.221673 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.221673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.222214 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.222216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.222584 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.222586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.222637 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.223215 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:07.223220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:07.225051 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260620 14:32:07.225268 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:07.220460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60885 (local address 127.0.98.62:42463)
0620 14:32:07.220642 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:07.220645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:07.220654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:32:07.220679 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:07.220680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:07.220691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:07.220756 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:07.221066 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.221601 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.222301 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.222490 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.222543 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.225178 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60885: BlockingRecv error: recv got EOF from 127.0.98.1:60885 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:07.356707 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:07.351578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:50647)
0620 14:32:07.351801 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:07.351815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:07.351923 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:07.352106 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:07.352107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:07.352295 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:07.352344 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.352346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.353040 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.353042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.353624 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.353626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.354018 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.354019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.354076 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.354625 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:07.354630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:07.356569 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
W20260620 14:32:07.356768 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:07.351728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50647 (local address 127.0.98.62:42463)
0620 14:32:07.351815 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:07.351817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:07.351896 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:32:07.352018 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:07.352019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:07.352037 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:07.352099 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:07.352453 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.352952 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.353727 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.353930 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.354017 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.356701 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50647: BlockingRecv error: recv got EOF from 127.0.98.2:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:07 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:07.485327 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:07.479351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:39055)
0620 14:32:07.479578 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:07.479590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:07.479672 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:07.479848 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:07.479849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:07.480097 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:07.480157 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.480159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.481522 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.481524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.482439 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:07.482443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.482911 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.482912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:07.482956 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.483407 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:07.483411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:07.485259 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
W20260620 14:32:07.485512 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:07.479447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39055 (local address 127.0.98.62:42463)
0620 14:32:07.479500 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:07.479503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:07.479675 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:32:07.479735 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:07.479737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:07.479752 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:07.479819 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:07.480243 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.481397 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.482543 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:07.482829 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:07.482907 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:07.485395 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39055: BlockingRecv error: recv got EOF from 127.0.98.3:39055 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:08.230482 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:08.225673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:49185)
0620 14:32:08.225791 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:08.225814 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:08.225863 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:08.226084 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:08.226086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:08.226261 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:08.226307 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.226309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.227028 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.227030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.227595 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.227597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.228023 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.228024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.228075 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.228558 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:08.228563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:08.230421 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260620 14:32:08.230623 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:08.225763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49185 (local address 127.0.98.62:42463)
0620 14:32:08.225916 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:08.225918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:08.225927 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:32:08.225950 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:08.225951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:08.225961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:08.226030 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:08.226442 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.226949 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.227680 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.227890 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.227962 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.230532 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49185: BlockingRecv error: recv got EOF from 127.0.98.1:49185 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:08.362200 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:08.357480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:57559)
0620 14:32:08.357660 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:08.357671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:08.357742 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:08.357882 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:08.357883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:08.358075 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:08.358120 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.358122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.358795 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.358796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.359332 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.359334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.359704 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.359705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.359764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.360233 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:08.360237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:08.362134 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
W20260620 14:32:08.362447 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:08.357577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57559 (local address 127.0.98.62:42463)
0620 14:32:08.357647 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:08.357649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:08.357721 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:32:08.357811 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:08.357813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:08.357824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:08.357878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:08.358202 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.358708 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.359422 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.359621 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.359695 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.362384 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57559: BlockingRecv error: recv got EOF from 127.0.98.2:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:08 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:08.490653 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:08.485862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:57343)
0620 14:32:08.486066 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:08.486080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:08.486175 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:08.486319 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:08.486320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:08.486512 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:08.486557 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.486560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.487247 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.487248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.487774 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:08.487775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.488145 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.488146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:08.488201 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.488666 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:08.488671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:08.490581 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
W20260620 14:32:08.490808 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:08.485993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57343 (local address 127.0.98.62:42463)
0620 14:32:08.486074 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:08.486077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:08.486155 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:32:08.486238 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:08.486239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:08.486254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:08.486301 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:08.486639 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.487149 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.487858 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:08.488070 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:08.488147 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:08.490703 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57343: BlockingRecv error: recv got EOF from 127.0.98.3:57343 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:09.235839 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:09.231085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:45841)
0620 14:32:09.231271 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:09.231285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:09.231373 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:09.231520 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:09.231521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:09.231686 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:09.231733 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.231735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.232372 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.232373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.232911 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.232913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.233358 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.233359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.233458 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.233991 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:09.234001 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:09.235736 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
W20260620 14:32:09.235888 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:09.231163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45841 (local address 127.0.98.62:42463)
0620 14:32:09.231319 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:09.231321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:09.231352 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:32:09.231436 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:09.231437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:09.231448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:09.231499 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:09.231801 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.232302 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.233011 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.233260 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.233353 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.235818 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45841: BlockingRecv error: recv got EOF from 127.0.98.1:45841 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:09.369836 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:09.362737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:43997)
0620 14:32:09.363094 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:09.363107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:09.363195 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:09.363415 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:09.363416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:09.363665 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:09.363741 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.363744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.365092 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.365095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.366138 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.366140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.366709 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.366710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.366768 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.367290 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:09.367295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:09.369733 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":243,"threads_started":1}
W20260620 14:32:09.370060 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:09.363047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43997 (local address 127.0.98.62:42463)
0620 14:32:09.363097 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:09.363100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:09.363202 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:32:09.363273 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:09.363275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:09.363291 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:09.363391 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:09.363821 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.364995 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.366242 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.366632 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.366697 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.369983 (+ 3286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43997: BlockingRecv error: recv got EOF from 127.0.98.2:43997 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:09 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:09.495944 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:09.491174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:37733)
0620 14:32:09.491362 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:09.491374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:09.491456 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:09.491616 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:09.491617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:09.491794 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:09.491840 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.491841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.492542 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.492543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.493069 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:09.493071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.493454 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.493455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:09.493512 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.494027 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:09.494032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:09.495886 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
W20260620 14:32:09.496032 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:09.491280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37733 (local address 127.0.98.62:42463)
0620 14:32:09.491355 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:09.491357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:09.491447 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:32:09.491517 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:09.491518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:09.491533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:09.491609 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:09.491918 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.492433 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.493169 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:09.493356 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:09.493409 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:09.495968 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37733: BlockingRecv error: recv got EOF from 127.0.98.3:37733 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:32:10 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:10 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:10.241204 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.236461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:42179)
0620 14:32:10.236638 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:10.236651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:10.236749 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:10.236942 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:10.236943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:10.237121 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:10.237168 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.237169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.237994 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.237995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.238503 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.238505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.238817 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.238817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.238854 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.239288 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:10.239292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:10.241147 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
W20260620 14:32:10.241360 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:10.236665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42179 (local address 127.0.98.62:42463)
0620 14:32:10.236808 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.236810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.236819 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:32:10.236850 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.236851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.236877 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.236935 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.237233 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.237900 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.238571 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.238755 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.238807 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.241253 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42179: BlockingRecv error: recv got EOF from 127.0.98.1:42179 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
Jun 20 14:32:10 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:32:10 dist-test-slave-tc2g krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:32:10.375120 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.370469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:50055)
0620 14:32:10.370599 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:10.370611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:10.370675 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:10.370862 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:10.370863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:10.371081 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:10.371153 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.371155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.371794 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.371795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.372409 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.372411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.372778 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.372779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.372822 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.373223 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:10.373242 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:10.375055 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260620 14:32:10.375221 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:10.370570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50055 (local address 127.0.98.62:42463)
0620 14:32:10.370612 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.370614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.370660 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:32:10.370736 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.370737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.370755 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.370834 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.371213 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.371727 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.372482 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.372715 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.372767 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.375146 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50055: BlockingRecv error: recv got EOF from 127.0.98.2:50055 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:32:10.414156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9112
I20260620 14:32:10.418596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9264
I20260620 14:32:10.422910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9398
I20260620 14:32:10.427116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9042
2026-06-20T14:32:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17411 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:32:10Z Disabled control of system clock
I20260620 14:32:10.454305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:32:10.535876 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:10.536091 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:10.536113 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:10.537230 9594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:10.537266 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:10.537281 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:10.537292 9594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:10.537304 9594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:10.538815 9594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:10.539016 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:10.539245 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:10.544312 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.544374 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:10.544412 9594 server_base.cc:1034] running on GCE node
W20260620 14:32:10.544376 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:10.544785 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:10.545079 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:10.546231 9594 hybrid_clock.cc:648] HybridClock initialized: now 1781965930546215 us; error 34 us; skew 500 ppm
I20260620 14:32:10.547508 9594 webserver.cc:460] Webserver started at http://127.0.98.62:40011/ using document root <none> and password file <none>
I20260620 14:32:10.547756 9594 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:10.547802 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:10.547943 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:10.549026 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "acf041d8708341b392dd5155c6bfef5b"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f5044b76ec0b5e90010f0d766982572d"
server_key_iv: "c23f41ec5417e7737d4d949a81cdb64f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.549471 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "acf041d8708341b392dd5155c6bfef5b"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f5044b76ec0b5e90010f0d766982572d"
server_key_iv: "c23f41ec5417e7737d4d949a81cdb64f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.550806 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:32:10.551525 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.551834 9594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:32:10.551929 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "acf041d8708341b392dd5155c6bfef5b"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f5044b76ec0b5e90010f0d766982572d"
server_key_iv: "c23f41ec5417e7737d4d949a81cdb64f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.552013 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:10.569741 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:10.570086 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:10.570235 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:10.574054 9594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39473
I20260620 14:32:10.574087 9660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39473 every 8 connection(s)
I20260620 14:32:10.574429 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:10.575078 9661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:10.577673 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b: Bootstrap starting.
I20260620 14:32:10.578256 9661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:10.578574 9661 log.cc:824] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:10.578601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9594
I20260620 14:32:10.578722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:32:10.578930 392 external_mini_cluster.cc:1428] Setting key df2e615cc62174ba2b25275c43a87d07
I20260620 14:32:10.579257 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b: No bootstrap required, opened a new log
I20260620 14:32:10.582109 9661 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260620 14:32:10.582273 9661 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:10.582305 9661 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf041d8708341b392dd5155c6bfef5b, State: Initialized, Role: FOLLOWER
I20260620 14:32:10.582427 9661 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260620 14:32:10.582504 9661 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:10.582533 9661 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:10.582569 9661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:10.583256 9661 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260620 14:32:10.583365 9661 leader_election.cc:302] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf041d8708341b392dd5155c6bfef5b; no voters:
I20260620 14:32:10.583534 9661 leader_election.cc:288] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:10.583588 9666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:10.583774 9666 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [term 1 LEADER]: Becoming Leader. State: Replica: acf041d8708341b392dd5155c6bfef5b, State: Running, Role: LEADER
I20260620 14:32:10.583782 9661 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:10.583915 9666 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } }
I20260620 14:32:10.584378 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf041d8708341b392dd5155c6bfef5b. Latest consensus state: current_term: 1 leader_uuid: "acf041d8708341b392dd5155c6bfef5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260620 14:32:10.584461 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:10.584718 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf041d8708341b392dd5155c6bfef5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf041d8708341b392dd5155c6bfef5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39473 } } }
I20260620 14:32:10.584827 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:10.584985 9673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:10.585831 9673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:32:10.585989 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.579442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.98.62:39473)
0620 14:32:10.579637 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.579646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.579679 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:32:10.580649 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.580659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.580739 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.580915 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.581117 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.582570 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.583679 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.584315 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.584769 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.585180 (+ 411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:10.585183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:10.585184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:10.585386 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:10.585448 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:10.585461 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:10.585523 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:10.585630 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:32:10.585710 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 14:32:10.587374 9673 catalog_manager.cc:1347] Generated new cluster ID: ffe118adca3b47489435c35d68843d05
I20260620 14:32:10.587426 9673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:10.594955 9673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:10.595640 9673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:10.604229 9673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf041d8708341b392dd5155c6bfef5b: Generated new TSK 0
I20260620 14:32:10.604441 9673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:32:10.606374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--builtin_ntp_servers=127.0.98.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:32:10.688344 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:10.688555 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:10.688578 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:10.689777 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:10.689821 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:10.689845 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:32:10.691159 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:10.691397 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:10.691711 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:10.697625 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.697635 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.697649 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:10.697873 9685 server_base.cc:1034] running on GCE node
I20260620 14:32:10.698081 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:10.698347 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:10.699538 9685 hybrid_clock.cc:648] HybridClock initialized: now 1781965930699516 us; error 31 us; skew 500 ppm
I20260620 14:32:10.700989 9685 webserver.cc:460] Webserver started at http://127.0.98.1:34345/ using document root <none> and password file <none>
I20260620 14:32:10.701248 9685 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:10.701319 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:10.701515 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:10.702683 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "514c12f4c5db4c7db823743bf85028e2"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a13f052fb0d6bc3ae8920daf778a06d1"
server_key_iv: "d4c7f9919ffb802e1df8fc7b27df009f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.703049 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "514c12f4c5db4c7db823743bf85028e2"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a13f052fb0d6bc3ae8920daf778a06d1"
server_key_iv: "d4c7f9919ffb802e1df8fc7b27df009f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.704370 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:32:10.705152 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.705346 9685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:32:10.705471 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "514c12f4c5db4c7db823743bf85028e2"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a13f052fb0d6bc3ae8920daf778a06d1"
server_key_iv: "d4c7f9919ffb802e1df8fc7b27df009f"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.705556 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:10.716548 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:10.716863 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:10.717020 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:10.717249 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:10.717657 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:10.717715 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.717757 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:10.717782 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.723726 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43149
I20260620 14:32:10.723768 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43149 every 8 connection(s)
I20260620 14:32:10.724099 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:10.729269 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.724825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44313 (local address 127.0.98.62:39473)
0620 14:32:10.724896 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.724898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.725137 (+ 239us) server_negotiation.cc:407] Connection header received
0620 14:32:10.725735 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.725736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.725750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.725795 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.726460 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.726961 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.727942 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.728138 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.728223 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.728995 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:10.728996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:10.728996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:10.729001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:10.729014 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:10.729018 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:10.729068 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:10.729164 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:32:10.729188 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:32:10.729554 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.724823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.1:44313)
0620 14:32:10.725012 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:10.725050 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:10.725631 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:10.725866 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:10.725869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:10.725878 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:10.726355 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.726359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.727062 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.727063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.727837 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.727839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.728232 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.728232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.728361 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.728853 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:10.728853 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:10.728881 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:10.728902 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:10.729077 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:10.729082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:10.729084 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:32:10.729167 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:32:10.729265 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260620 14:32:10.730059 9813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260620 14:32:10.730134 9813 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:10.730270 9813 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260620 14:32:10.730711 9625 ts_manager.cc:194] Registered new tserver with Master: 514c12f4c5db4c7db823743bf85028e2 (127.0.98.1:43149)
I20260620 14:32:10.731335 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44313
I20260620 14:32:10.732473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9685
I20260620 14:32:10.732549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:10.732755 392 external_mini_cluster.cc:1428] Setting key 8b152f059afc9610c2b827855da02cfb
I20260620 14:32:10.733613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--builtin_ntp_servers=127.0.98.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:32:10.813318 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:10.813558 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:10.813582 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:10.814779 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:10.814845 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:10.814886 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:32:10.816198 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:10.816459 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:10.816757 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:32:10.821724 9816 server_base.cc:1034] running on GCE node
W20260620 14:32:10.821707 9824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.821681 9822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.821693 9821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:10.822106 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:10.822350 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:10.823520 9816 hybrid_clock.cc:648] HybridClock initialized: now 1781965930823513 us; error 27 us; skew 500 ppm
I20260620 14:32:10.824818 9816 webserver.cc:460] Webserver started at http://127.0.98.2:37931/ using document root <none> and password file <none>
I20260620 14:32:10.825060 9816 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:10.825133 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:10.825286 9816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:10.826483 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "565c09e79037440bbe1cbe6cf7f5616a"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f0793b2da22fbad93dc426e1d27e9a81"
server_key_iv: "72df44abc74c060ca6c0892bf363231e"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.826848 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "565c09e79037440bbe1cbe6cf7f5616a"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f0793b2da22fbad93dc426e1d27e9a81"
server_key_iv: "72df44abc74c060ca6c0892bf363231e"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.828148 9816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:32:10.828852 9830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.829030 9816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:32:10.829126 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "565c09e79037440bbe1cbe6cf7f5616a"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "f0793b2da22fbad93dc426e1d27e9a81"
server_key_iv: "72df44abc74c060ca6c0892bf363231e"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.829213 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:10.840052 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:10.840360 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:10.840508 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:10.840730 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:10.841055 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:10.841109 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.841149 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:10.841177 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.846735 9816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42213
I20260620 14:32:10.846781 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42213 every 8 connection(s)
I20260620 14:32:10.847086 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:32:10.847983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9816
I20260620 14:32:10.848078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:32:10.848284 392 external_mini_cluster.cc:1428] Setting key da531107880590f317ee0ccbf854b0ab
I20260620 14:32:10.849377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--builtin_ntp_servers=127.0.98.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:32:10.852937 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.847744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35331 (local address 127.0.98.62:39473)
0620 14:32:10.847860 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.847863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.848087 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:32:10.848766 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.848767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.848778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.848868 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.849797 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.850270 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.851393 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.851728 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.851799 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.852613 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:10.852615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:10.852615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:10.852620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:10.852638 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:10.852643 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:10.852702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:10.852795 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:32:10.852823 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:32:10.853112 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.847701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.2:35331)
0620 14:32:10.847926 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:10.847975 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:10.848655 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:10.848953 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:10.848958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:10.848968 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:10.849704 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.849711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.850357 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.850358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.851273 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.851275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.851817 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.851818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.851920 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.852468 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:10.852469 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:10.852499 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:10.852521 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:10.852716 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:10.852720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:10.852723 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:32:10.852777 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:32:10.852862 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260620 14:32:10.853745 9944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260620 14:32:10.853847 9944 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:10.854027 9944 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260620 14:32:10.854435 9625 ts_manager.cc:194] Registered new tserver with Master: 565c09e79037440bbe1cbe6cf7f5616a (127.0.98.2:42213)
I20260620 14:32:10.854808 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35331
W20260620 14:32:10.931190 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:10.931403 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:10.931424 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:10.932701 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:10.932745 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:10.932771 9947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:32:10.934103 9947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:10.934306 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:10.934554 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:10.939549 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.939523 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:10.939533 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:10.939771 9947 server_base.cc:1034] running on GCE node
I20260620 14:32:10.940019 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:10.940284 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:10.941526 9947 hybrid_clock.cc:648] HybridClock initialized: now 1781965930941507 us; error 30 us; skew 500 ppm
I20260620 14:32:10.942754 9947 webserver.cc:460] Webserver started at http://127.0.98.3:38813/ using document root <none> and password file <none>
I20260620 14:32:10.943007 9947 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:10.943056 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:10.943222 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:10.944294 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "d684e44b4c364fbdbe203f7b11d75a94"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a09df7983e2882006511e103ab3761a6"
server_key_iv: "7ed4142d946b31a5460230a6ebefcf84"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.944656 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d684e44b4c364fbdbe203f7b11d75a94"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a09df7983e2882006511e103ab3761a6"
server_key_iv: "7ed4142d946b31a5460230a6ebefcf84"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.946005 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:32:10.946710 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.946890 9947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:32:10.946987 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "d684e44b4c364fbdbe203f7b11d75a94"
format_stamp: "Formatted at 2026-06-20 14:32:10 on dist-test-slave-tc2g"
server_key: "a09df7983e2882006511e103ab3761a6"
server_key_iv: "7ed4142d946b31a5460230a6ebefcf84"
server_key_version: "encryptionkey@0"
I20260620 14:32:10.947067 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:10.976069 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:10.976413 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:10.976573 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:10.976814 9947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:10.977173 9947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:10.977226 9947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.977267 9947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:10.977308 9947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:10.982936 9947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38911
I20260620 14:32:10.982980 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38911 every 8 connection(s)
I20260620 14:32:10.983310 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:32:10.984722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 9947
I20260620 14:32:10.984798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:32:10.984980 392 external_mini_cluster.cc:1428] Setting key 8ab7ddb21402a82a4f3bcb29811d4b8c
I20260620 14:32:10.988770 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.984095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59427 (local address 127.0.98.62:39473)
0620 14:32:10.984137 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.984139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.984403 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:32:10.985011 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.985013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.985025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.985080 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.985960 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.986459 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.987447 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.987652 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.987734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.988501 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:10.988503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:10.988503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:10.988508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:10.988523 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:10.988527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:10.988586 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:10.988669 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:32:10.988694 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:32:10.989001 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:10.984085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39473 (local address 127.0.98.3:59427)
0620 14:32:10.984282 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:10.984316 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:10.984911 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:10.985150 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:10.985154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:10.985166 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:10.985865 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.985871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.986553 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.986554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.987349 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:10.987351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:10.987742 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:10.987743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:10.987888 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:10.988359 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:10.988359 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:10.988386 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:10.988408 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:10.988596 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:10.988600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:10.988603 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:32:10.988663 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:32:10.988762 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260620 14:32:10.989507 10075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39473
I20260620 14:32:10.989600 10075 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:10.989737 10075 heartbeater.cc:511] Master 127.0.98.62:39473 requested a full tablet report, sending...
I20260620 14:32:10.990073 9625 ts_manager.cc:194] Registered new tserver with Master: d684e44b4c364fbdbe203f7b11d75a94 (127.0.98.3:38911)
I20260620 14:32:10.990391 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59427
I20260620 14:32:10.996842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:32:10.998801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9685
W20260620 14:32:10.998802 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:10.998207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.0.98.62:39473)
0620 14:32:10.998392 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:10.998394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:10.998402 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:32:10.998430 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:10.998430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:10.998438 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:10.998486 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:10.998721 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43026: BlockingRecv error: recv got EOF from 127.0.0.1:43026 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
I20260620 14:32:11.003327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9816
I20260620 14:32:11.007773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9947
I20260620 14:32:11.012107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 9594
2026-06-20T14:32:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (584 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:32:11Z Disabled control of system clock
I20260620 14:32:11.038475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:32:11.118718 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:11.118932 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:11.118953 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:11.120034 10086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:11.120069 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:11.120081 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:11.120093 10086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:11.120105 10086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:11.121459 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:11.121692 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:11.121930 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:11.126714 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.126772 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.126847 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:11.127116 10086 server_base.cc:1034] running on GCE node
I20260620 14:32:11.127378 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:11.127683 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:11.128844 10086 hybrid_clock.cc:648] HybridClock initialized: now 1781965931128826 us; error 26 us; skew 500 ppm
I20260620 14:32:11.130169 10086 webserver.cc:460] Webserver started at http://127.0.98.62:35613/ using document root <none> and password file <none>
I20260620 14:32:11.130438 10086 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:11.130512 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:11.130677 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:11.131800 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "437590a42ddf44f2ab0e0221280f9354"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "dc8feca188590d48da083e54a173fae9"
server_key_iv: "14fc643cd7209b51b4f5f3297b4c7de1"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.132207 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "437590a42ddf44f2ab0e0221280f9354"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "dc8feca188590d48da083e54a173fae9"
server_key_iv: "14fc643cd7209b51b4f5f3297b4c7de1"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.133548 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:32:11.134280 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.134440 10086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:32:11.134539 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "437590a42ddf44f2ab0e0221280f9354"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "dc8feca188590d48da083e54a173fae9"
server_key_iv: "14fc643cd7209b51b4f5f3297b4c7de1"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.134620 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:11.137975 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:11.138249 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:11.138391 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:11.142093 10086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37891
I20260620 14:32:11.142131 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37891 every 8 connection(s)
I20260620 14:32:11.142452 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:11.143072 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:11.145805 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354: Bootstrap starting.
I20260620 14:32:11.146373 10153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:11.146662 10153 log.cc:824] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:11.147286 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354: No bootstrap required, opened a new log
I20260620 14:32:11.148794 10153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 14:32:11.148975 10153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:11.149029 10153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437590a42ddf44f2ab0e0221280f9354, State: Initialized, Role: FOLLOWER
I20260620 14:32:11.149179 10153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 14:32:11.149276 10153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:11.149340 10153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:11.149390 10153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:11.149936 10153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 14:32:11.150056 10153 leader_election.cc:302] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 437590a42ddf44f2ab0e0221280f9354; no voters:
I20260620 14:32:11.150243 10153 leader_election.cc:288] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:11.150285 10156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:11.150461 10156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [term 1 LEADER]: Becoming Leader. State: Replica: 437590a42ddf44f2ab0e0221280f9354, State: Running, Role: LEADER
I20260620 14:32:11.150501 10153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:11.150556 10156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 14:32:11.150987 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 437590a42ddf44f2ab0e0221280f9354. Latest consensus state: current_term: 1 leader_uuid: "437590a42ddf44f2ab0e0221280f9354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260620 14:32:11.151074 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:11.151216 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "437590a42ddf44f2ab0e0221280f9354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "437590a42ddf44f2ab0e0221280f9354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260620 14:32:11.151310 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:11.151763 10165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:11.152281 10165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:32:11.152467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10086
I20260620 14:32:11.152570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:32:11.152767 392 external_mini_cluster.cc:1428] Setting key f6a5c68ba2732762f022147e8b59d0c3
I20260620 14:32:11.153868 10165 catalog_manager.cc:1347] Generated new cluster ID: d6047cf3c4e243578c73fc0f7e2ba528
I20260620 14:32:11.153918 10165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:11.155400 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.153196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.98.62:37891)
0620 14:32:11.153373 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:11.153383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:11.153500 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:32:11.153687 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:11.153695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:11.153739 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:11.153871 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:11.154931 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:11.154935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:11.154936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:11.154941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:11.154986 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:11.155001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:11.155037 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:11.155169 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:32:11.155227 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260620 14:32:11.160903 10165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:11.161054 10165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:11.166281 10165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 437590a42ddf44f2ab0e0221280f9354: Generated new TSK 0
I20260620 14:32:11.166468 10165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:32:11.173559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:32:11.254500 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:11.254726 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:11.254765 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:11.255918 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:11.255980 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:11.256021 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:32:11.257380 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:11.257691 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:11.257984 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:11.262912 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.262948 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.262944 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:11.263072 10177 server_base.cc:1034] running on GCE node
I20260620 14:32:11.263363 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:11.263635 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:11.264793 10177 hybrid_clock.cc:648] HybridClock initialized: now 1781965931264785 us; error 31 us; skew 500 ppm
I20260620 14:32:11.269668 10177 webserver.cc:460] Webserver started at http://127.0.98.1:40657/ using document root <none> and password file <none>
I20260620 14:32:11.269939 10177 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:11.269989 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:11.270157 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:11.271248 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f573c54f5ff40d39b165a178a8daf2d"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "f7c4e1385a215370edbbc53ecf55c16a"
server_key_iv: "20101ca31caa174083fe848fce33cfc2"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.271864 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f573c54f5ff40d39b165a178a8daf2d"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "f7c4e1385a215370edbbc53ecf55c16a"
server_key_iv: "20101ca31caa174083fe848fce33cfc2"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.273247 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:32:11.274115 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.274271 10177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:32:11.274353 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "0f573c54f5ff40d39b165a178a8daf2d"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "f7c4e1385a215370edbbc53ecf55c16a"
server_key_iv: "20101ca31caa174083fe848fce33cfc2"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.274407 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:11.277747 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:11.278025 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:11.278164 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:11.278365 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:11.278666 10177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:11.278716 10177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.278755 10177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:11.278781 10177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.284590 10177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37979
I20260620 14:32:11.284629 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37979 every 8 connection(s)
I20260620 14:32:11.284972 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:11.287835 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.285736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60633 (local address 127.0.98.62:37891)
0620 14:32:11.285786 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:11.285789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:11.286018 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:32:11.286527 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:11.286528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:11.286535 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:11.286588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:11.287659 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:11.287660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:11.287660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:11.287664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:11.287677 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:11.287681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:11.287727 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:11.287771 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:32:11.287782 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:32:11.288022 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.285737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.1:60633)
0620 14:32:11.285915 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:11.285950 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:11.286413 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:11.286641 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:11.286644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:11.286651 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:11.287519 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:11.287520 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:11.287548 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:11.287577 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:11.287723 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:11.287726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:11.287728 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:32:11.287778 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:32:11.287824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 14:32:11.288453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10177
I20260620 14:32:11.288578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:11.288614 10305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260620 14:32:11.288693 10305 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:11.288789 392 external_mini_cluster.cc:1428] Setting key ddeecb12700b795ac791ef14e57feb40
I20260620 14:32:11.288802 10305 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260620 14:32:11.289354 10117 ts_manager.cc:194] Registered new tserver with Master: 0f573c54f5ff40d39b165a178a8daf2d (127.0.98.1:37979)
I20260620 14:32:11.290028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:32:11.372488 10308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:11.372697 10308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:11.372720 10308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:11.373924 10308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:11.373966 10308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:11.373992 10308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:32:11.375381 10308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:11.375620 10308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:11.375907 10308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:11.381135 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.381141 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.381145 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:11.381798 10308 server_base.cc:1034] running on GCE node
I20260620 14:32:11.382031 10308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:11.382237 10308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:11.383402 10308 hybrid_clock.cc:648] HybridClock initialized: now 1781965931383397 us; error 38 us; skew 500 ppm
I20260620 14:32:11.384728 10308 webserver.cc:460] Webserver started at http://127.0.98.2:41879/ using document root <none> and password file <none>
I20260620 14:32:11.384975 10308 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:11.385046 10308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:11.385186 10308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:11.386343 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "947ac2bfdce34dfb9b8a3ea48e3a41a3"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "974d1a0c9387045c99218e19f86db3bd"
server_key_iv: "95f2b64ed492321d61fe264f0e85b926"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.386729 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "947ac2bfdce34dfb9b8a3ea48e3a41a3"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "974d1a0c9387045c99218e19f86db3bd"
server_key_iv: "95f2b64ed492321d61fe264f0e85b926"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.388069 10308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:32:11.388743 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.389110 10308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:32:11.389213 10308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "947ac2bfdce34dfb9b8a3ea48e3a41a3"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "974d1a0c9387045c99218e19f86db3bd"
server_key_iv: "95f2b64ed492321d61fe264f0e85b926"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.389288 10308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:11.392799 10308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:11.393075 10308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:11.393217 10308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:11.393465 10308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:11.393769 10308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:11.393822 10308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.393878 10308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:11.393906 10308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.399505 10308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41005
I20260620 14:32:11.399545 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41005 every 8 connection(s)
I20260620 14:32:11.399865 10308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:32:11.402874 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.400575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54101 (local address 127.0.98.62:37891)
0620 14:32:11.400624 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:11.400626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:11.400869 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:32:11.401393 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:11.401430 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:11.401439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:11.401489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:11.402659 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:11.402660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:11.402660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:11.402664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:11.402680 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:11.402685 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:11.402740 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:11.402801 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:32:11.402813 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:32:11.403071 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.400566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.2:54101)
0620 14:32:11.400750 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:11.400783 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:11.401276 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:11.401628 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:11.401631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:11.401638 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:11.402506 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:11.402507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:11.402533 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:11.402571 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:11.402741 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:11.402744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:11.402746 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:32:11.402797 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:32:11.402843 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260620 14:32:11.403585 10436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260620 14:32:11.403663 10436 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:11.403771 10436 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260620 14:32:11.404091 10117 ts_manager.cc:194] Registered new tserver with Master: 947ac2bfdce34dfb9b8a3ea48e3a41a3 (127.0.98.2:41005)
I20260620 14:32:11.404484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10308
I20260620 14:32:11.404600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:32:11.404776 392 external_mini_cluster.cc:1428] Setting key bd673026b9ad2e76b30ba433d2479997
I20260620 14:32:11.405623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--builtin_ntp_servers=127.0.98.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:32:11.490108 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:11.490301 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:11.490324 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:11.491408 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:11.491441 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:11.491466 10439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:32:11.492801 10439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:11.493026 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:11.493315 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:11.498085 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.498117 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:11.498219 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:11.498318 10439 server_base.cc:1034] running on GCE node
I20260620 14:32:11.498510 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:11.498786 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:11.499966 10439 hybrid_clock.cc:648] HybridClock initialized: now 1781965931499945 us; error 38 us; skew 500 ppm
I20260620 14:32:11.501154 10439 webserver.cc:460] Webserver started at http://127.0.98.3:34963/ using document root <none> and password file <none>
I20260620 14:32:11.501379 10439 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:11.501463 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:11.501621 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:11.502707 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ab224f80efa414fb01c6bfbe8475c2a"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "91c90d61b2338791b39722af50d80582"
server_key_iv: "c053534d71f435733654f0acd5a86f50"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.503062 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ab224f80efa414fb01c6bfbe8475c2a"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "91c90d61b2338791b39722af50d80582"
server_key_iv: "c053534d71f435733654f0acd5a86f50"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.504439 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:32:11.505112 10453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.505285 10439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:32:11.505378 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "5ab224f80efa414fb01c6bfbe8475c2a"
format_stamp: "Formatted at 2026-06-20 14:32:11 on dist-test-slave-tc2g"
server_key: "91c90d61b2338791b39722af50d80582"
server_key_iv: "c053534d71f435733654f0acd5a86f50"
server_key_version: "encryptionkey@0"
I20260620 14:32:11.505479 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:11.508682 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:11.508934 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:11.509078 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:11.509279 10439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:11.509624 10439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:11.509675 10439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.509722 10439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:11.509752 10439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:11.515476 10439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42179
I20260620 14:32:11.515511 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42179 every 8 connection(s)
I20260620 14:32:11.515810 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:32:11.518522 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.516515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36105 (local address 127.0.98.62:37891)
0620 14:32:11.516566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:11.516569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:11.516770 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:32:11.517187 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:11.517188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:11.517193 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:11.517248 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:11.518325 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:11.518327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:11.518327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:32:11.518331 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:11.518345 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:32:11.518348 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:11.518397 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:11.518458 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:32:11.518473 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:32:11.518719 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:11.516468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.98.3:36105)
0620 14:32:11.516646 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:11.516683 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:11.517109 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:11.517304 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:11.517307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:11.517314 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:11.518190 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:32:11.518191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:11.518217 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:32:11.518243 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:11.518405 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:11.518409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:11.518411 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:32:11.518467 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:32:11.518511 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260620 14:32:11.519160 10567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37891
I20260620 14:32:11.519249 10567 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:11.519390 10567 heartbeater.cc:511] Master 127.0.98.62:37891 requested a full tablet report, sending...
I20260620 14:32:11.519711 10117 ts_manager.cc:194] Registered new tserver with Master: 5ab224f80efa414fb01c6bfbe8475c2a (127.0.98.3:42179)
I20260620 14:32:11.520147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10439
I20260620 14:32:11.520247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:32:11.520430 392 external_mini_cluster.cc:1428] Setting key bbe3274b9819adbb99bd08857af22fa8
I20260620 14:32:11.521251 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:32:11.522974 10175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:32:11.522524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.98.62:37891)
0620 14:32:11.522560 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:11.522561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:11.522693 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:32:11.522744 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:11.522745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:11.522748 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:11.522794 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:11.522892 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49668: BlockingRecv error: recv got EOF from 127.0.0.1:49668 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:32:11.523074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10177
I20260620 14:32:11.527593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10308
I20260620 14:32:11.532220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10439
I20260620 14:32:11.536291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10086
2026-06-20T14:32:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:32:11 dist-test-slave-tc2g krb5kdc[10577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:32:11 dist-test-slave-tc2g krb5kdc[10577](info): set up 2 sockets
Jun 20 14:32:11 dist-test-slave-tc2g krb5kdc[10577](info): commencing operation
krb5kdc: starting...
W20260620 14:32:12.574389 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:32:12 dist-test-slave-tc2g krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:32:12.692993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:12.772739 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:12.773020 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:12.773061 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:12.774314 10590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:12.774387 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:12.774412 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:12.774441 10590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:12.774472 10590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:12.775955 10590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:12.776252 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:12.776484 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:12.781478 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:12.781553 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:12.781559 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:12.781733 10590 server_base.cc:1034] running on GCE node
I20260620 14:32:12.781961 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:12.782249 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:12.783412 10590 hybrid_clock.cc:648] HybridClock initialized: now 1781965932783400 us; error 54 us; skew 500 ppm
Jun 20 14:32:12 dist-test-slave-tc2g krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:12.785931 10590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:32:12.786366 10590 webserver.cc:460] Webserver started at http://127.0.98.62:44989/ using document root <none> and password file <none>
I20260620 14:32:12.786602 10590 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:12.786674 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:12.786823 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:12.787904 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "be9a9c0b221c4fc3b643831d37e85976"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "f75f465201d7f733c9cceee937be6ca3"
server_key_iv: "083233823bd5980fa287dfc8c622c32d"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.788269 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "be9a9c0b221c4fc3b643831d37e85976"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "f75f465201d7f733c9cceee937be6ca3"
server_key_iv: "083233823bd5980fa287dfc8c622c32d"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.789606 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:32:12.790292 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:12.790453 10590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:12.790561 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "be9a9c0b221c4fc3b643831d37e85976"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "f75f465201d7f733c9cceee937be6ca3"
server_key_iv: "083233823bd5980fa287dfc8c622c32d"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.790637 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:12.798480 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:12.800729 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:12.800889 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:12.804738 10590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37455
I20260620 14:32:12.804770 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37455 every 8 connection(s)
I20260620 14:32:12.805119 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:12.806128 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:12.807734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10590
I20260620 14:32:12.807847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:32:12.808039 392 external_mini_cluster.cc:1428] Setting key dd756c782bfddd19e3e6c4c31d944689
I20260620 14:32:12.808635 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: Bootstrap starting.
I20260620 14:32:12.809242 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:12.809602 10658 log.cc:824] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:12.810266 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: No bootstrap required, opened a new log
I20260620 14:32:12.811952 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260620 14:32:12.812115 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:12.812165 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9a9c0b221c4fc3b643831d37e85976, State: Initialized, Role: FOLLOWER
I20260620 14:32:12.812296 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260620 14:32:12.812402 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:12.812461 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:12.812517 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:12.813036 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260620 14:32:12.813156 10658 leader_election.cc:302] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be9a9c0b221c4fc3b643831d37e85976; no voters:
I20260620 14:32:12.813345 10658 leader_election.cc:288] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:12.813390 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:12.813597 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [term 1 LEADER]: Becoming Leader. State: Replica: be9a9c0b221c4fc3b643831d37e85976, State: Running, Role: LEADER
I20260620 14:32:12.813627 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:12.813822 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
Jun 20 14:32:12 dist-test-slave-tc2g krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:12.814999 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be9a9c0b221c4fc3b643831d37e85976. Latest consensus state: current_term: 1 leader_uuid: "be9a9c0b221c4fc3b643831d37e85976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } } }
I20260620 14:32:12.815119 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be9a9c0b221c4fc3b643831d37e85976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be9a9c0b221c4fc3b643831d37e85976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } } }
I20260620 14:32:12.815312 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:12.815315 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976 [sys.catalog]: This master's current role is: LEADER
W20260620 14:32:12.815492 10678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:32:12.815548 10678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:32:12.817571 10680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:12.817760 10680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:32:12.819190 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:12.808596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.98.62:37455)
0620 14:32:12.808783 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:12.808792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:12.808828 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:32:12.809253 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:12.809260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:12.809305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:12.809528 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:12.810162 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:12.811101 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:12.811877 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:12.812370 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:12.812490 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:12.815529 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:12.815533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:12.815533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:12.815547 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:12.817413 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:12.817890 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:12.817894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:12.817894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:12.817998 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:12.818229 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:12.818230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:12.818231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:12.818514 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:12.818681 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:12.818807 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:32:12.818954 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260620 14:32:12.819499 10680 catalog_manager.cc:1347] Generated new cluster ID: 4d91e4a16ff842ee80628cbdd7a3c438
I20260620 14:32:12.819545 10680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:12.840811 10680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:12.841359 10680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:12.851325 10680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: Generated new TSK 0
I20260620 14:32:12.851550 10680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:32:12.880486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:12.960390 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:12.960588 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:12.960610 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:12.961719 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:12.961757 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:12.961783 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:32:12.963124 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:12.963292 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:12.963466 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:12.968225 10692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:12.968181 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:12.968179 10689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:12.968557 10684 server_base.cc:1034] running on GCE node
I20260620 14:32:12.968760 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:12.968950 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:12.970083 10684 hybrid_clock.cc:648] HybridClock initialized: now 1781965932970068 us; error 31 us; skew 500 ppm
Jun 20 14:32:12 dist-test-slave-tc2g krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:12.972615 10684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:32:12.973074 10684 webserver.cc:460] Webserver started at http://127.0.98.1:45647/ using document root <none> and password file <none>
I20260620 14:32:12.973268 10684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:12.973308 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:12.973402 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:12.974438 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "2891125505084c9ab57398995be8519d"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "822fe68e6688372e34f148a769cfbda0"
server_key_iv: "186a4367ed4ca709000a055ea6eeef5e"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.974737 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2891125505084c9ab57398995be8519d"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "822fe68e6688372e34f148a769cfbda0"
server_key_iv: "186a4367ed4ca709000a055ea6eeef5e"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.976011 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:32:12.976684 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:12.976866 10684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:32:12.976931 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "2891125505084c9ab57398995be8519d"
format_stamp: "Formatted at 2026-06-20 14:32:12 on dist-test-slave-tc2g"
server_key: "822fe68e6688372e34f148a769cfbda0"
server_key_iv: "186a4367ed4ca709000a055ea6eeef5e"
server_key_version: "encryptionkey@0"
I20260620 14:32:12.977020 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:12.992901 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:12.995312 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:12.995476 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:12.995709 10684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:12.996026 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:12.996079 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:12.996124 10684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:12.996150 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.002059 10684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35063
I20260620 14:32:13.002104 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35063 every 8 connection(s)
I20260620 14:32:13.002421 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:13.005368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10684
I20260620 14:32:13.005491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:13.005698 392 external_mini_cluster.cc:1428] Setting key a805cca44ca21d041edb628d43e5978a
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:13.012579 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.003182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60589 (local address 127.0.98.62:37455)
0620 14:32:13.003243 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.003245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.003512 (+ 267us) server_negotiation.cc:407] Connection header received
0620 14:32:13.004045 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.004046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.004058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.004118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:13.005174 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.005885 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.006928 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.007156 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.007245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.009750 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:13.009752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:13.009752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:13.009761 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:13.011154 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.011578 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.011579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.011580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.011645 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.011903 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.011904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.011905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.012035 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:13.012110 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.012383 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.012481 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:32:13.012777 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.003152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37455 (local address 127.0.98.1:60589)
0620 14:32:13.003359 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.003398 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.003932 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.004198 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.004201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.004812 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:13.005075 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.005081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.005978 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.005978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.006826 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.006828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.007256 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.007257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.007427 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.007985 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:13.007996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:13.009656 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:13.011260 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.011267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.011270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.011500 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.011721 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.011722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.011722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.011836 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.012101 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:13.012108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:13.012252 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.012363 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.012533 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260620 14:32:13.013273 10813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37455
I20260620 14:32:13.013365 10813 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:13.013649 10813 heartbeater.cc:511] Master 127.0.98.62:37455 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:32:13.014286 10622 ts_manager.cc:194] Registered new tserver with Master: 2891125505084c9ab57398995be8519d (127.0.98.1:35063)
I20260620 14:32:13.015014 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:32:13.031747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:13.113633 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:13.113937 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:13.113968 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:13.115159 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:13.115202 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:13.115229 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:32:13.116703 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:13.116920 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:13.117107 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:13.122018 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:13.121994 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:13.121994 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:13.122023 10818 server_base.cc:1034] running on GCE node
I20260620 14:32:13.122493 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:13.122735 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:13.123903 10818 hybrid_clock.cc:648] HybridClock initialized: now 1781965933123894 us; error 29 us; skew 500 ppm
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:13.126526 10818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:32:13.126935 10818 webserver.cc:460] Webserver started at http://127.0.98.2:34039/ using document root <none> and password file <none>
I20260620 14:32:13.127123 10818 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:13.127163 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:13.127252 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:13.128249 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "186155c3db8e4f488044a8e531b99771"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "8040ec33c3060de481db55476149430c"
server_key_iv: "ddf610192770442d55c9f4cb190da210"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.128535 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "186155c3db8e4f488044a8e531b99771"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "8040ec33c3060de481db55476149430c"
server_key_iv: "ddf610192770442d55c9f4cb190da210"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.129956 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:32:13.130671 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.130844 10818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:32:13.130951 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "186155c3db8e4f488044a8e531b99771"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "8040ec33c3060de481db55476149430c"
server_key_iv: "ddf610192770442d55c9f4cb190da210"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.131026 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:13.144968 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:13.146924 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:13.147104 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:13.147346 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:13.147665 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:13.147720 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.147763 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:13.147790 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.153350 10818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39431
I20260620 14:32:13.153465 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39431 every 8 connection(s)
I20260620 14:32:13.153782 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:32:13.156704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10818
I20260620 14:32:13.156805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:32:13.156987 392 external_mini_cluster.cc:1428] Setting key aa6ac619e92c27ceabf17f6d4b636926
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:13.164326 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.154523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56141 (local address 127.0.98.62:37455)
0620 14:32:13.154631 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.154634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.154804 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:32:13.155387 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.155388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.155400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.155452 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:13.156472 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.157243 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.158758 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.158979 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.159036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.161392 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:13.161393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:13.161394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:13.161402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:13.162908 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.163353 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.163354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.163355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.163412 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.163674 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.163675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.163676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.163791 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:13.163867 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.164146 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.164230 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:32:13.164549 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.154484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37455 (local address 127.0.98.2:56141)
0620 14:32:13.154670 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.154721 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.155266 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.155522 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.155526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.156174 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:13.156368 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.156373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.157348 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.157349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.158668 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.158671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.159067 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.159069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.159227 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.159829 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:13.159837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:13.161279 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:13.163012 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.163019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.163021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.163259 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.163478 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.163479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.163480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.163605 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.163857 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:13.163864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:13.164012 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.164125 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.164296 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:32:13.165037 10947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37455
I20260620 14:32:13.165166 10947 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:13.165354 10947 heartbeater.cc:511] Master 127.0.98.62:37455 requested a full tablet report, sending...
I20260620 14:32:13.165798 10622 ts_manager.cc:194] Registered new tserver with Master: 186155c3db8e4f488044a8e531b99771 (127.0.98.2:39431)
I20260620 14:32:13.166236 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:32:13.181862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:13.262205 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:13.262537 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:13.262584 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:13.263767 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:13.263839 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:13.263885 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:32:13.265350 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:13.265666 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:13.265894 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:13.270799 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:13.270799 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:13.270813 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:13.271099 10952 server_base.cc:1034] running on GCE node
I20260620 14:32:13.271311 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:13.271601 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:13.272768 10952 hybrid_clock.cc:648] HybridClock initialized: now 1781965933272752 us; error 47 us; skew 500 ppm
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:13.275381 10952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:32:13.275780 10952 webserver.cc:460] Webserver started at http://127.0.98.3:45069/ using document root <none> and password file <none>
I20260620 14:32:13.275991 10952 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:13.276034 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:13.276129 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:32:13.277096 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "bb38002c14489bd47a4c76991c602187"
server_key_iv: "d0bb526acc00678e8639e038be618994"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.277395 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "bb38002c14489bd47a4c76991c602187"
server_key_iv: "d0bb526acc00678e8639e038be618994"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.278664 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:32:13.279327 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.279500 10952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.279582 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-tc2g"
server_key: "bb38002c14489bd47a4c76991c602187"
server_key_iv: "d0bb526acc00678e8639e038be618994"
server_key_version: "encryptionkey@0"
I20260620 14:32:13.279639 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:13.291332 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:13.293243 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:13.293354 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:13.293602 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:13.293918 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:13.293951 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.294005 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:13.294037 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.299568 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46467
I20260620 14:32:13.299664 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46467 every 8 connection(s)
I20260620 14:32:13.299962 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:13.306506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 10952
I20260620 14:32:13.306664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:32:13.306936 392 external_mini_cluster.cc:1428] Setting key 91122a063e62b1fe50665cb3364a0bad
I20260620 14:32:13.308966 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.300679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53441 (local address 127.0.98.62:37455)
0620 14:32:13.300748 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.300750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.300954 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:32:13.301447 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.301448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.301457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.301503 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:13.302452 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.302942 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.303902 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.304090 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.304143 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.306343 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:13.306344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:13.306345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:13.306354 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:13.307919 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.308278 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.308278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.308279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.308326 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.308497 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.308498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.308498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.308595 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:13.308642 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.308842 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.308901 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:32:13.309264 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.300653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37455 (local address 127.0.98.3:53441)
0620 14:32:13.300827 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.300869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.301344 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.301583 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.301586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.302180 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:13.302371 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.302376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.303035 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.303035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.303825 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.303827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.304163 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.304164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.304257 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.304808 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:13.304817 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:13.306240 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:13.307989 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.307994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.307995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.308213 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.308366 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:13.308366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:13.308366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:13.308450 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:13.308633 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:13.308635 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:13.308742 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.308817 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.308930 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260620 14:32:13.309810 11081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37455
I20260620 14:32:13.309921 11081 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:13.310111 11081 heartbeater.cc:511] Master 127.0.98.62:37455 requested a full tablet report, sending...
I20260620 14:32:13.310513 10622 ts_manager.cc:194] Registered new tserver with Master: 3e8a2b8583564a87ba4f3dabb1eb1bdc (127.0.98.3:46467)
I20260620 14:32:13.311038 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53441
I20260620 14:32:13.318676 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:32:13.326447 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.319984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.0.98.62:37455)
0620 14:32:13.320062 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.320065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.320161 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:32:13.320250 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.320250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.320261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.320317 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:13.320867 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.321362 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.322126 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.322330 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.322403 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.323922 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:13.323924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:13.323925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:13.323934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:13.325409 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.325764 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.325764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.325764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.325813 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.325959 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.325959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.325959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.326097 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:13.326170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.326252 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.326310 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:32:13.327953 10622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:32:13.329603 10622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:32:13.335906 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39431 (local address 127.0.0.1:49774)
0620 14:32:13.331729 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.331767 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.331836 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.332170 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.332171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.332173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:13.332249 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.332251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.333606 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.333607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.334707 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.334708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.335595 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.335596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.335684 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.335688 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.335762 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.335827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":22,"thread_start_us":62,"threads_started":1}
I20260620 14:32:13.335913 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.98.1:35063)
0620 14:32:13.331977 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.331982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.331992 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:32:13.332032 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.332034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.332048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.332133 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:13.332304 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.333139 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.334694 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.335461 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.335562 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.335641 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.335785 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.335825 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260620 14:32:13.336032 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.0.98.2:39431)
0620 14:32:13.331977 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.331982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.331993 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:32:13.332034 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.332036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.332066 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.332327 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:13.332399 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.333487 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.334774 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.335460 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.335810 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.335873 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.335895 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.335940 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260620 14:32:13.336153 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35063 (local address 127.0.0.1:48644)
0620 14:32:13.331825 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.331834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.331876 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.332146 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.332149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.332151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:13.332217 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.332219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.333244 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.333245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.334607 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.334610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.335558 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.335559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.335666 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.335685 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.335759 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.336096 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260620 14:32:13.336809 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f (DEFAULT_TABLE table=test-table [id=55510d42ae8149b692e20ac79f78b131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:13.336933 10747 tablet_service.cc:1467] Processing CreateTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f (DEFAULT_TABLE table=test-table [id=55510d42ae8149b692e20ac79f78b131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:13.337224 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6800f088a77b4fe8a1c1de3dc576e20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:13.337286 10747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6800f088a77b4fe8a1c1de3dc576e20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:13.340020 11096 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Bootstrap starting.
I20260620 14:32:13.340607 11096 tablet_bootstrap.cc:654] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:13.340760 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46467 (local address 127.0.0.1:44664)
0620 14:32:13.331942 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.331952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.331986 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.332213 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.332214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.332215 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:13.332272 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.332273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.334786 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.334787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.335492 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.335494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.340501 (+ 5007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.340502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.340570 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.340575 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.340641 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.340686 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":19,"thread_start_us":44,"threads_started":1}
I20260620 14:32:13.340773 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.331797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.0.98.3:46467)
0620 14:32:13.331977 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.331982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.331993 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:32:13.332037 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.332039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.332052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.332140 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:13.332350 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.333485 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.335632 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.340400 (+ 4768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.340489 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.340546 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.340656 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.340697 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260620 14:32:13.341317 11096 log.cc:824] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:13.341375 11015 tablet_service.cc:1467] Processing CreateTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f (DEFAULT_TABLE table=test-table [id=55510d42ae8149b692e20ac79f78b131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:13.341614 11095 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Bootstrap starting.
I20260620 14:32:13.341737 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6800f088a77b4fe8a1c1de3dc576e20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:13.342134 11096 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: No bootstrap required, opened a new log
I20260620 14:32:13.342209 11096 ts_tablet_manager.cc:1397] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:32:13.342284 11095 tablet_bootstrap.cc:654] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:13.342819 11095 log.cc:824] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:13.343585 11095 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: No bootstrap required, opened a new log
I20260620 14:32:13.343700 11095 ts_tablet_manager.cc:1397] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:32:13.344058 11096 raft_consensus.cc:348] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.344229 11096 raft_consensus.cc:374] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:13.344272 11096 raft_consensus.cc:729] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186155c3db8e4f488044a8e531b99771, State: Initialized, Role: FOLLOWER
I20260620 14:32:13.344383 11096 consensus_queue.cc:260] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.344691 11096 ts_tablet_manager.cc:1428] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:32:13.344693 10947 heartbeater.cc:503] Master 127.0.98.62:37455 was elected leader, sending a full tablet report...
I20260620 14:32:13.345592 11100 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Bootstrap starting.
I20260620 14:32:13.345904 11095 raft_consensus.cc:348] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.346093 11095 raft_consensus.cc:374] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:13.346149 11095 raft_consensus.cc:729] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2891125505084c9ab57398995be8519d, State: Initialized, Role: FOLLOWER
I20260620 14:32:13.346283 11100 tablet_bootstrap.cc:654] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:13.346307 11095 consensus_queue.cc:260] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.346635 10813 heartbeater.cc:503] Master 127.0.98.62:37455 was elected leader, sending a full tablet report...
I20260620 14:32:13.346725 11095 ts_tablet_manager.cc:1428] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:32:13.346982 11100 log.cc:824] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:13.347798 11100 tablet_bootstrap.cc:492] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: No bootstrap required, opened a new log
I20260620 14:32:13.347884 11100 ts_tablet_manager.cc:1397] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:32:13.349195 11100 raft_consensus.cc:348] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.349351 11100 raft_consensus.cc:374] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:13.349437 11100 raft_consensus.cc:729] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8a2b8583564a87ba4f3dabb1eb1bdc, State: Initialized, Role: FOLLOWER
I20260620 14:32:13.349568 11100 consensus_queue.cc:260] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.349813 11100 ts_tablet_manager.cc:1428] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:32:13.349855 11081 heartbeater.cc:503] Master 127.0.98.62:37455 was elected leader, sending a full tablet report...
W20260620 14:32:13.404384 10948 tablet.cc:2369] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:32:13.503150 10814 tablet.cc:2369] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:32:13.550565 11082 tablet.cc:2369] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:32:13.573598 11099 raft_consensus.cc:482] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:32:13.573760 11099 raft_consensus.cc:504] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.574173 11099 leader_election.cc:288] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2891125505084c9ab57398995be8519d (127.0.98.1:35063), 3e8a2b8583564a87ba4f3dabb1eb1bdc (127.0.98.3:46467)
I20260620 14:32:13.577072 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.574404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46467 (local address 127.0.98.2:44777)
0620 14:32:13.574463 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.574475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.574524 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.574746 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.574747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.574748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:13.574820 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.574822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.575462 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.575463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.576231 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.576232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.576795 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.576879 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.576882 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.576958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.577003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:32:13.577072 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.574519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44777 (local address 127.0.98.3:46467)
0620 14:32:13.574589 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.574591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.574600 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:32:13.574632 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.574633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.574645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.574736 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:13.574892 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.575372 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576328 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.576707 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576795 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.576805 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.576966 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.577003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:32:13.577172 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.574436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54683 (local address 127.0.98.1:35063)
0620 14:32:13.574482 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.574484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.574767 (+ 283us) server_negotiation.cc:407] Connection header received
0620 14:32:13.574819 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.574820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.574832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.574908 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:13.575068 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.575537 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576468 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.576824 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576908 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.576917 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.577075 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.577112 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:32:13.577175 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.574368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35063 (local address 127.0.98.2:54683)
0620 14:32:13.574691 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.574699 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:13.574767 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:13.574922 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:13.574922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:13.574924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:13.574991 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.574994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.575630 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.575631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.576373 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:13.576375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.576910 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.576911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:13.576986 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.576995 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:32:13.577057 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:32:13.577097 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":79,"thread_start_us":138,"threads_started":1}
I20260620 14:32:13.577489 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6800f088a77b4fe8a1c1de3dc576e20f" candidate_uuid: "186155c3db8e4f488044a8e531b99771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" is_pre_election: true
I20260620 14:32:13.577525 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6800f088a77b4fe8a1c1de3dc576e20f" candidate_uuid: "186155c3db8e4f488044a8e531b99771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2891125505084c9ab57398995be8519d" is_pre_election: true
I20260620 14:32:13.577668 10767 raft_consensus.cc:2454] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186155c3db8e4f488044a8e531b99771 in term 0.
I20260620 14:32:13.577668 11035 raft_consensus.cc:2454] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186155c3db8e4f488044a8e531b99771 in term 0.
I20260620 14:32:13.577922 10834 leader_election.cc:302] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186155c3db8e4f488044a8e531b99771, 3e8a2b8583564a87ba4f3dabb1eb1bdc; no voters:
I20260620 14:32:13.578055 11099 raft_consensus.cc:2781] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:32:13.578119 11099 raft_consensus.cc:482] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:32:13.578148 11099 raft_consensus.cc:3037] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:13.578807 11099 raft_consensus.cc:504] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.578953 11099 leader_election.cc:288] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [CANDIDATE]: Term 1 election: Requested vote from peers 2891125505084c9ab57398995be8519d (127.0.98.1:35063), 3e8a2b8583564a87ba4f3dabb1eb1bdc (127.0.98.3:46467)
I20260620 14:32:13.579116 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6800f088a77b4fe8a1c1de3dc576e20f" candidate_uuid: "186155c3db8e4f488044a8e531b99771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2891125505084c9ab57398995be8519d"
I20260620 14:32:13.579106 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6800f088a77b4fe8a1c1de3dc576e20f" candidate_uuid: "186155c3db8e4f488044a8e531b99771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc"
I20260620 14:32:13.579188 11035 raft_consensus.cc:3037] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:13.579188 10767 raft_consensus.cc:3037] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:13.580138 10767 raft_consensus.cc:2454] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186155c3db8e4f488044a8e531b99771 in term 1.
I20260620 14:32:13.580149 11035 raft_consensus.cc:2454] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186155c3db8e4f488044a8e531b99771 in term 1.
I20260620 14:32:13.580293 10836 leader_election.cc:302] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186155c3db8e4f488044a8e531b99771, 2891125505084c9ab57398995be8519d; no voters:
I20260620 14:32:13.580397 11099 raft_consensus.cc:2781] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:13.580618 11099 raft_consensus.cc:686] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 1 LEADER]: Becoming Leader. State: Replica: 186155c3db8e4f488044a8e531b99771, State: Running, Role: LEADER
I20260620 14:32:13.580737 11099 consensus_queue.cc:237] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } }
I20260620 14:32:13.581624 10621 catalog_manager.cc:5495] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 186155c3db8e4f488044a8e531b99771 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "186155c3db8e4f488044a8e531b99771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "186155c3db8e4f488044a8e531b99771" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20260620 14:32:13.618719 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.616110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.98.2:39431)
0620 14:32:13.616168 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.616169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.616179 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:32:13.616300 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.616301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.616322 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.616406 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:32:13.616560 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.617032 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.617893 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.618089 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.618174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.618354 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:32:13.618460 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:32:13.618570 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.618587 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.618607 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:32:13.620666 11035 raft_consensus.cc:1261] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 1 FOLLOWER]: Refusing update from remote peer 186155c3db8e4f488044a8e531b99771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:13.620675 10767 raft_consensus.cc:1261] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 1 FOLLOWER]: Refusing update from remote peer 186155c3db8e4f488044a8e531b99771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:13.621057 11099 consensus_queue.cc:1035] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2891125505084c9ab57398995be8519d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:13.621215 11105 consensus_queue.cc:1035] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a2b8583564a87ba4f3dabb1eb1bdc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:13.623787 11109 mvcc.cc:205] Tried to move back new op lower bound from 7298932464107167744 to 7298932463947259904. Current Snapshot: MvccSnapshot[applied={T|T < 7298932464107167744 or (T in {7298932464107167744})}]
I20260620 14:32:13.625470 11110 mvcc.cc:205] Tried to move back new op lower bound from 7298932464107167744 to 7298932463947259904. Current Snapshot: MvccSnapshot[applied={T|T < 7298932464107167744 or (T in {7298932464107167744})}]
I20260620 14:32:13.625792 10622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:13.625885 10622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:13.626616 10622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be9a9c0b221c4fc3b643831d37e85976: Sending DeleteTablet for 3 replicas of tablet 6800f088a77b4fe8a1c1de3dc576e20f
I20260620 14:32:13.626921 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49774
I20260620 14:32:13.627002 10747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48644
I20260620 14:32:13.627156 11015 tablet_service.cc:1514] Processing DeleteTablet for tablet 6800f088a77b4fe8a1c1de3dc576e20f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44664
I20260620 14:32:13.627264 11118 tablet_replica.cc:330] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d: stopping tablet replica
I20260620 14:32:13.627396 11118 raft_consensus.cc:2229] T 6800f088a77b4fe8a1c1de3dc576e20f P 2891125505084c9ab57398995be8519d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:32:13.627413 11119 tablet_replica.cc:330] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: stopping tablet replica
I20260620 14:32:13.627492 11119 raft_consensus.cc:2229] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:32:13.627585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10684
I20260620 14:32:13.627604 11119 raft_consensus.cc:2258] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:13.628572 11119 ts_tablet_manager.cc:1905] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:32:13.630028 11119 ts_tablet_manager.cc:1918] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:32:13.630120 11119 log.cc:1196] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/6800f088a77b4fe8a1c1de3dc576e20f
I20260620 14:32:13.630375 11119 ts_tablet_manager.cc:1939] T 6800f088a77b4fe8a1c1de3dc576e20f P 3e8a2b8583564a87ba4f3dabb1eb1bdc: Deleting consensus metadata
I20260620 14:32:13.630762 10606 catalog_manager.cc:4841] TS 3e8a2b8583564a87ba4f3dabb1eb1bdc (127.0.98.3:46467): tablet 6800f088a77b4fe8a1c1de3dc576e20f (table test-table [id=55510d42ae8149b692e20ac79f78b131]) successfully deleted
I20260620 14:32:13.634006 11117 tablet_replica.cc:330] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: stopping tablet replica
I20260620 14:32:13.634132 11117 raft_consensus.cc:2229] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:32:13.634236 11117 raft_consensus.cc:2258] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:13.634651 11117 ts_tablet_manager.cc:1905] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:32:13.635198 10608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35063 (error 108)
I20260620 14:32:13.635658 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.635546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35063 (local address 127.0.0.1:48658)
0620 14:32:13.635591 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.635623 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:32:13.635769 10608 catalog_manager.cc:4584] TS 2891125505084c9ab57398995be8519d (127.0.98.1:35063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6800f088a77b4fe8a1c1de3dc576e20f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35063: connect: Connection refused (error 111)
I20260620 14:32:13.636080 11117 ts_tablet_manager.cc:1918] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:32:13.636168 11117 log.cc:1196] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/6800f088a77b4fe8a1c1de3dc576e20f
I20260620 14:32:13.636191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10818
I20260620 14:32:13.636430 11117 ts_tablet_manager.cc:1939] T 6800f088a77b4fe8a1c1de3dc576e20f P 186155c3db8e4f488044a8e531b99771: Deleting consensus metadata
I20260620 14:32:13.641304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10952
I20260620 14:32:13.641449 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.641265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39431 (local address 127.0.0.1:49792)
0620 14:32:13.641333 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:13.641385 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:32:13.646345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 10590
2026-06-20T14:32:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2110 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[11123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[11123](info): set up 2 sockets
Jun 20 14:32:13 dist-test-slave-tc2g krb5kdc[11123](info): commencing operation
krb5kdc: starting...
W20260620 14:32:14.678407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:32:14 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:32:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:32:14.835700 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpSwtz3/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:32:17.198 UTC [11153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:32:17.198 UTC [11153] LOG: listening on IPv4 address "127.0.98.20", port 41679
2026-06-20 14:32:17.201 UTC [11153] LOG: listening on Unix socket "/tmp/.s.PGSQL.41679"
2026-06-20 14:32:17.231 UTC [11156] LOG: database system was shut down at 2026-06-20 14:32:15 UTC
2026-06-20 14:32:17.234 UTC [11153] LOG: database system is ready to accept connections
I20260620 14:32:18.190299 392 mini_postgres.cc:96] Postgres bound to 41679
2026-06-20 14:32:18.196 UTC [11164] FATAL: database "slave" does not exist
127.0.98.20:41679 - accepting connections
I20260620 14:32:18.197126 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:32:18.212236 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:32:18.667258 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:32:18.667344 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin
2026-06-20 14:32:18,948 [I] DB FLAVOR :POSTGRES
2026-06-20 14:32:18,949 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:32:18,949 [I] Checking connection..
2026-06-20 14:32:18,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:32:19,215 [I] Checking connection passed.
2026-06-20 14:32:19,215 [I] --------- Verifying version history table ---------
2026-06-20 14:32:19,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:19,470 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:32:19,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:19,702 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:32:19,702 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:32:19,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:32:19.950 UTC [11228] WARNING: there is no transaction in progress
2026-06-20 14:32:19,956 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:32:19,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:20,212 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:32:20,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:32:20,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:32:20,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:32:20,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tc2g.c.gcp-upstream.internal','N') ;"
2026-06-20 14:32:20,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:32:21,173 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:32:21,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:32:21,408 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:32:21,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:32:21,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:32:21,899 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:32:21,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:32:22.850 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.865 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.881 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.896 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.912 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.932 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.992 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:22.999 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.014 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.025 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.036 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.049 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.059 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.065 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.071 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23.077 UTC [11336] WARNING: there is no transaction in progress
2026-06-20 14:32:23,666 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:32:23,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:32:23,897 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:32:23,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:32:24,158 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:32:24,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:32:24,389 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:32:24,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:32:24,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:32:24,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tc2g.c.gcp-upstream.internal';"
2026-06-20 14:32:25,115 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:32:25,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:32:25,348 [I] DB_PATCHES have already been applied
I20260620 14:32:25.354578 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpSwtz3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:32:25.354647 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:32:25.358307 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37505
Jun 20, 2026 2:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:32:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:32:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:32:26.485103 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.288s user 0.005s sys 0.005s
Jun 20, 2026 2:32:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:32:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:32:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:32:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:32:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:32:47.875931 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:32:47 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965967, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:32:47.953346 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:32:47.962239 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:32:48.778504 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:32:48.778591 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms
I20260620 14:32:48.778609 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41679
2026-06-20 14:32:48,956 [I] DB FLAVOR :POSTGRES
2026-06-20 14:32:48,956 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:32:48,956 [I] Checking connection
2026-06-20 14:32:48,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:32:49,197 [I] connection success
2026-06-20 14:32:49,198 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:32:49,198 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:32:49,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:32:49,424 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:32:49,425 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:32:49,425 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:32:49,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:32:49,696 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:32:50.139204 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:32:50.319103 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:32:50.362007 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:32:50 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965967, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:50.772284 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:32:50.772398 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:32:50.772418 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:32:50.775954 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11516
I20260620 14:32:50.776057 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47461
Jun 20, 2026 2:32:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:32:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:32:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:32:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:32:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:32:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:32:53.021641 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.068s user 0.001s sys 0.160s
I20260620 14:32:53.021742 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:32:53.021813 392 mini_ranger_kms.cc:401] 127.0.98.20:47461/kms/v1/keys
Jun 20 14:32:53 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:32:53 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:32:53 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:32:56 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:32:56.222414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42571
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42571
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:56.306223 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:56.306455 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:56.306492 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:56.307825 11567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:56.307868 11567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:56.307883 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:56.307894 11567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:56.307906 11567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:56.309505 11567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42571
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:56.309754 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:56.309986 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:56.314853 11573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:56.314812 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:56.314807 11572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:56.315055 11567 server_base.cc:1034] running on GCE node
I20260620 14:32:56.315295 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:56.315574 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:56.316730 11567 hybrid_clock.cc:648] HybridClock initialized: now 1781965976316710 us; error 37 us; skew 500 ppm
Jun 20 14:32:56 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:56.322731 11567 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:32:56.323261 11567 webserver.cc:460] Webserver started at http://127.0.98.62:33897/ using document root <none> and password file <none>
I20260620 14:32:56.323504 11567 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:56.323565 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:56.323709 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:56 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:56.368762 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "04f4f494124c4cfea437a9725b7c880d"
format_stamp: "Formatted at 2026-06-20 14:32:56 on dist-test-slave-tc2g"
server_key: "2c9625ef75e1801e4236542cd0c4b87c"
server_key_iv: "ad02672db9fd154570ccbce3d0a072ac"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:56.369239 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "04f4f494124c4cfea437a9725b7c880d"
format_stamp: "Formatted at 2026-06-20 14:32:56 on dist-test-slave-tc2g"
server_key: "2c9625ef75e1801e4236542cd0c4b87c"
server_key_iv: "ad02672db9fd154570ccbce3d0a072ac"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:56.371124 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:32:56.382615 11582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:56.382855 11567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:32:56.382951 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "04f4f494124c4cfea437a9725b7c880d"
format_stamp: "Formatted at 2026-06-20 14:32:56 on dist-test-slave-tc2g"
server_key: "2c9625ef75e1801e4236542cd0c4b87c"
server_key_iv: "ad02672db9fd154570ccbce3d0a072ac"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:56.383069 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:56.402307 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:56.404595 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:56.404796 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:56.409970 11567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42571
I20260620 14:32:56.409992 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42571 every 8 connection(s)
I20260620 14:32:56.410373 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:56.411039 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:56.413324 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: Bootstrap starting.
I20260620 14:32:56.413955 11635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:56.414225 11635 log.cc:824] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:56.414888 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: No bootstrap required, opened a new log
I20260620 14:32:56.416860 11635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260620 14:32:56.416993 11635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:56.417016 11635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f4f494124c4cfea437a9725b7c880d, State: Initialized, Role: FOLLOWER
I20260620 14:32:56.417162 11635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260620 14:32:56.417240 11635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:56.417263 11635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:56.417330 11635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:56.417870 11635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260620 14:32:56.417999 11635 leader_election.cc:302] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04f4f494124c4cfea437a9725b7c880d; no voters:
I20260620 14:32:56.418138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 11567
I20260620 14:32:56.418170 11635 leader_election.cc:288] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:56.418241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:32:56.418222 11638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:56.418413 11638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [term 1 LEADER]: Becoming Leader. State: Replica: 04f4f494124c4cfea437a9725b7c880d, State: Running, Role: LEADER
I20260620 14:32:56.418437 392 external_mini_cluster.cc:1428] Setting key 06bc0fc55fcbaa34681c7e06faee9256
I20260620 14:32:56.418433 11635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:56.418503 11638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260620 14:32:56.418943 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f4f494124c4cfea437a9725b7c880d. Latest consensus state: current_term: 1 leader_uuid: "04f4f494124c4cfea437a9725b7c880d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } } }
I20260620 14:32:56.419070 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f4f494124c4cfea437a9725b7c880d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f4f494124c4cfea437a9725b7c880d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } } }
I20260620 14:32:56.419100 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:56.419198 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:56.420292 11645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:56.420439 11635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:32:56.420913 11645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:32:56 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:56.430109 11645 catalog_manager.cc:1347] Generated new cluster ID: d12675821cb6403fa38204e68c3dd308
I20260620 14:32:56.430238 11645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:56.444478 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:56.418935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.0.98.62:42571)
0620 14:32:56.420049 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:56.420060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:56.420101 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:32:56.420226 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:56.420235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:56.420295 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:56.420439 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:56.421331 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:56.423261 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:56.424350 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:56.424962 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:56.425089 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:56.428635 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:56.428692 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:56.428692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:56.428810 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:56.435350 (+ 6540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:56.438672 (+ 3322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:56.438676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:56.438676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:56.438778 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:56.443457 (+ 4679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:56.443458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:56.443459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:56.443786 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:56.443934 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:56.444070 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:32:56.444233 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":71,"threads_started":1}
I20260620 14:32:56.445497 11645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:56.446376 11645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:56.460058 11645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: Generated new TSK 0
I20260620 14:32:56.460364 11645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:32:57.158506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:57.248551 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:57.248800 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:57.248840 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:57.250094 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:57.250136 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:57.250164 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:32:57.251694 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:57.252012 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:57.252271 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:57.257362 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:57.257368 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:57.257652 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:57.258340 11680 server_base.cc:1034] running on GCE node
I20260620 14:32:57.258584 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:57.258852 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:57.260015 11680 hybrid_clock.cc:648] HybridClock initialized: now 1781965977259990 us; error 40 us; skew 500 ppm
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:57.262939 11680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:32:57.263509 11680 webserver.cc:460] Webserver started at http://127.0.98.1:44025/ using document root <none> and password file <none>
I20260620 14:32:57.263777 11680 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:57.263847 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:57.264026 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:57.278031 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4223c8f4df743e4860fd0c6a891a1e1"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "e2e6ff24b45730d41c5719ada268355a"
server_key_iv: "1b04c805779fe3d8f3c25c6bc3deb310"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.278550 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4223c8f4df743e4860fd0c6a891a1e1"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "e2e6ff24b45730d41c5719ada268355a"
server_key_iv: "1b04c805779fe3d8f3c25c6bc3deb310"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.281805 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:32:57.297609 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.297833 11680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 14:32:57.297932 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "f4223c8f4df743e4860fd0c6a891a1e1"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "e2e6ff24b45730d41c5719ada268355a"
server_key_iv: "1b04c805779fe3d8f3c25c6bc3deb310"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.298059 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:57.313815 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:57.316272 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:57.316421 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:57.316623 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:57.317015 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:57.317061 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.317116 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:57.317133 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.324146 11680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45467
I20260620 14:32:57.324564 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:57.327412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 11680
I20260620 14:32:57.327535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:57.327740 392 external_mini_cluster.cc:1428] Setting key c8ccd50e9e7d1afe367d338788421f70
I20260620 14:32:57.334935 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45467 every 8 connection(s)
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:32:57.349946 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.336019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42571 (local address 127.0.98.1:47423)
0620 14:32:57.338766 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.338818 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.339488 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.339951 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.339956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.340226 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:57.340510 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.340517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.341544 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.341545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.342817 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.342819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.343373 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.343373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.343511 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.344110 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:57.344121 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:57.345770 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:57.347964 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.347972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.347975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.348287 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.348566 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.348567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.348568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.348760 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.349126 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:57.349133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:57.349334 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.349510 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.349697 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":96,"threads_started":1}
I20260620 14:32:57.350394 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.336332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47423 (local address 127.0.98.62:42571)
0620 14:32:57.339694 (+ 3362us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.339696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.339705 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:32:57.339732 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.339733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.339746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.339814 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:57.340623 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.341439 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.342919 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.343220 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.343294 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.345878 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:57.345880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:57.345880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:57.345888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:57.347870 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.348383 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.348384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.348385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.348450 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.348850 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.348851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.348852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.349000 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:57.349074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.350210 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.350299 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3328,"thread_start_us":76,"threads_started":1}
I20260620 14:32:57.350772 11811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42571
I20260620 14:32:57.350869 11811 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:57.351060 11811 heartbeater.cc:511] Master 127.0.98.62:42571 requested a full tablet report, sending...
I20260620 14:32:57.351599 11599 ts_manager.cc:194] Registered new tserver with Master: f4223c8f4df743e4860fd0c6a891a1e1 (127.0.98.1:45467)
I20260620 14:32:57.352428 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:32:57.370610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:32:57.524461 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:57.524729 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:57.524761 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:57.526379 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:57.526438 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:57.526476 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:32:57.528543 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:57.528798 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:57.529045 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:57.535943 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:57.538581 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:57.538640 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:57.539559 11817 server_base.cc:1034] running on GCE node
I20260620 14:32:57.539788 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:57.540069 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:57.543332 11817 hybrid_clock.cc:648] HybridClock initialized: now 1781965977543292 us; error 50 us; skew 500 ppm
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:57.546602 11817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:32:57.547127 11817 webserver.cc:460] Webserver started at http://127.0.98.2:43311/ using document root <none> and password file <none>
I20260620 14:32:57.547386 11817 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:57.547434 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:57.547554 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:57.561966 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "e323e06bb77142bbbae984e9987a8c91"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "fe57d1c66008ba7fb2642c7b70fa5482"
server_key_iv: "7da66a459ede345ff2a9674303f1b222"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.562551 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e323e06bb77142bbbae984e9987a8c91"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "fe57d1c66008ba7fb2642c7b70fa5482"
server_key_iv: "7da66a459ede345ff2a9674303f1b222"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.566715 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:32:57.582259 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.582517 11817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:32:57.582624 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "e323e06bb77142bbbae984e9987a8c91"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "fe57d1c66008ba7fb2642c7b70fa5482"
server_key_iv: "7da66a459ede345ff2a9674303f1b222"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.582746 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:57.596323 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:57.598630 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:57.598820 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:57.599087 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:57.599462 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:57.599514 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.599552 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:57.599578 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.606838 11817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34017
I20260620 14:32:57.607262 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:32:57.608058 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34017 every 8 connection(s)
I20260620 14:32:57.615288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 11817
I20260620 14:32:57.615394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:32:57.615579 392 external_mini_cluster.cc:1428] Setting key d47dfbec4a229055984e06515ad07ea8
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:57.621047 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.608996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41213 (local address 127.0.98.62:42571)
0620 14:32:57.609044 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.609046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.609390 (+ 344us) server_negotiation.cc:407] Connection header received
0620 14:32:57.610707 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.610708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.610722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.610753 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:57.611568 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.612384 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.613957 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.614261 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.614305 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.617613 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:57.617614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:57.617615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:57.617623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:57.619456 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.619957 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.619958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.619958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.620015 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.620280 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.620281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.620281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.620412 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:57.620462 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.620856 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.620945 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:32:57.621827 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.608725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42571 (local address 127.0.98.2:41213)
0620 14:32:57.609264 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.609308 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.610562 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.610933 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.610938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.611184 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:57.611456 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.611462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.612502 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.612503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.613847 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.613850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.614397 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.614398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.614537 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.615166 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:57.615179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:57.617508 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:57.619555 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.619562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.619565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.619869 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.620080 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.620081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.620081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.620208 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.620499 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:57.620504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:57.620697 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.621394 (+ 697us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.621588 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":79,"threads_started":1}
I20260620 14:32:57.622548 11949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42571
I20260620 14:32:57.622642 11949 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:57.622843 11949 heartbeater.cc:511] Master 127.0.98.62:42571 requested a full tablet report, sending...
I20260620 14:32:57.623239 11599 ts_manager.cc:194] Registered new tserver with Master: e323e06bb77142bbbae984e9987a8c91 (127.0.98.2:34017)
I20260620 14:32:57.623741 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41213
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:32:57.661890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:57.784909 11954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:57.785395 11954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:57.785527 11954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:57.787395 11954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:57.787523 11954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:57.787614 11954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:32:57.789983 11954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:32:57.790431 11954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:57.790741 11954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:57.797478 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:57.797569 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:57.797621 11954 server_base.cc:1034] running on GCE node
W20260620 14:32:57.797736 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:57.797943 11954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:57.798194 11954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:57.799381 11954 hybrid_clock.cc:648] HybridClock initialized: now 1781965977799360 us; error 32 us; skew 500 ppm
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:57.802634 11954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:32:57.803105 11954 webserver.cc:460] Webserver started at http://127.0.98.3:42425/ using document root <none> and password file <none>
I20260620 14:32:57.803364 11954 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:57.803427 11954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:57.803560 11954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:57.817564 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "107f2f6c39a83eb459c1faed2c1d1998"
server_key_iv: "352c527c62c3863b6774454de801b1d9"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.818145 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "107f2f6c39a83eb459c1faed2c1d1998"
server_key_iv: "352c527c62c3863b6774454de801b1d9"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.822228 11954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:32:57.836879 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.837056 11954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:32:57.837143 11954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7"
format_stamp: "Formatted at 2026-06-20 14:32:57 on dist-test-slave-tc2g"
server_key: "107f2f6c39a83eb459c1faed2c1d1998"
server_key_iv: "352c527c62c3863b6774454de801b1d9"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:57.837239 11954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:57.858799 11954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:57.860975 11954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:57.861141 11954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:57.861400 11954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:57.861802 11954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:57.861857 11954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.861899 11954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:57.861924 11954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:57.868712 11954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43983
I20260620 14:32:57.869132 11954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:32:57.871261 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43983 every 8 connection(s)
Jun 20 14:32:57 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:32:57.879123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 11954
I20260620 14:32:57.879235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:32:57.879423 392 external_mini_cluster.cc:1428] Setting key 3a5505461382149e73ebd0c7063733b2
I20260620 14:32:57.881927 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.872251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41969 (local address 127.0.98.62:42571)
0620 14:32:57.872293 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.872295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.873763 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 14:32:57.874305 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.874306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.874319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.874367 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:57.874935 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.875445 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.876462 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.876691 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.876768 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.878839 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:57.878841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:57.878841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:57.878849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:57.880488 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.880964 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.880965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.880966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.881033 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.881282 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.881283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.881284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.881391 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:57.881515 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.881771 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.881842 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:32:57.882145 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.872251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42571 (local address 127.0.98.3:41969)
0620 14:32:57.873622 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.873682 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.874187 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.874437 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.874441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.874640 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:57.874844 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.874848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.875556 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.875557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.876356 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.876358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.876772 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.876773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.876898 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.877495 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:57.877504 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:57.878739 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:57.880573 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.880580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.880582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.880880 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.881108 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:57.881108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:57.881109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:57.881200 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:57.881501 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:57.881507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:57.881643 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.881751 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.881889 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":1284,"threads_started":1}
I20260620 14:32:57.882632 12083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42571
I20260620 14:32:57.882747 12083 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:57.882953 12083 heartbeater.cc:511] Master 127.0.98.62:42571 requested a full tablet report, sending...
I20260620 14:32:57.885634 11599 ts_manager.cc:194] Registered new tserver with Master: 3d24959c1e5a45bfbba43ba5c7dd55b7 (127.0.98.3:43983)
I20260620 14:32:57.886108 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41969
I20260620 14:32:57.891618 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:32:57.900537 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.893261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.0.98.62:42571)
0620 14:32:57.893305 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.893308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.893571 (+ 263us) server_negotiation.cc:407] Connection header received
0620 14:32:57.893634 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.893635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.893653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.893707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:57.894286 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.894926 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.895654 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.895851 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.895923 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.897327 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:57.897328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:57.897329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:57.897336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:57.899055 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.899512 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.899514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.899514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.899572 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:57.899764 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:57.899765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:57.899765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:57.899884 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:57.899970 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.900368 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.900441 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:32:57.902468 11599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:32:57.904600 11599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:32:57.912279 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.907353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34017 (local address 127.0.0.1:39266)
0620 14:32:57.907495 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.907535 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.907593 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.908763 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.908767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.908769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:57.908842 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.908845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.909917 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.909918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.911052 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.911055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.912015 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.912016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.912094 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.912109 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.912151 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.912198 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":95,"threads_started":1}
I20260620 14:32:57.914868 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.907928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.0.98.3:43983)
0620 14:32:57.908106 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.908111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.909633 (+ 1522us) server_negotiation.cc:407] Connection header received
0620 14:32:57.909675 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.909678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.909691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.909752 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:57.911211 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.912244 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.913524 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.914275 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.914347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.914419 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.914752 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.914792 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260620 14:32:57.914683 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.907802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43983 (local address 127.0.0.1:48980)
0620 14:32:57.909516 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.909526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.909578 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.909824 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.909825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.909826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:57.909862 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.909863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.912328 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.912329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.913372 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.913375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.914454 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.914455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.914516 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.914520 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.914570 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.914608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1699,"thread_start_us":78,"threads_started":1}
I20260620 14:32:57.917038 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.907967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.0.98.2:34017)
0620 14:32:57.908571 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.908576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.908586 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:32:57.908630 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.908632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.908657 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.908761 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:57.908967 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.909856 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.911144 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.911916 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.916790 (+ 4874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.916877 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.916903 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.916945 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":79,"threads_started":1}
I20260620 14:32:57.917349 12004 tablet_service.cc:1467] Processing CreateTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 (DEFAULT_TABLE table=test-table [id=9044e3d1930a4ac29ca09d760defc834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:57.917768 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f4ef37c8643c484d07fc7f49a87d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:57.917860 11879 tablet_service.cc:1467] Processing CreateTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 (DEFAULT_TABLE table=test-table [id=9044e3d1930a4ac29ca09d760defc834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:57.918149 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f4ef37c8643c484d07fc7f49a87d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:57.918979 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.907596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45467 (local address 127.0.0.1:36512)
0620 14:32:57.908856 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:57.908866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:57.908907 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:57.915286 (+ 6379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:57.915288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:57.915290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:57.915328 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.915330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.916572 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.916573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.917699 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:57.917702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.918739 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.918741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:57.918803 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.918807 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:32:57.918861 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:32:57.918899 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":78,"threads_started":1}
I20260620 14:32:57.919282 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:57.908190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.98.1:45467)
0620 14:32:57.915044 (+ 6854us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:57.915049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:57.915063 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:32:57.915100 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:57.915102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:57.915118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:57.915212 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:57.915411 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.916461 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.917801 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:57.918573 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:57.918647 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:57.918701 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:57.919169 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 14:32:57.919213 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6574,"thread_start_us":63,"threads_started":1}
I20260620 14:32:57.919881 11737 tablet_service.cc:1467] Processing CreateTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 (DEFAULT_TABLE table=test-table [id=9044e3d1930a4ac29ca09d760defc834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:57.920225 11737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f4ef37c8643c484d07fc7f49a87d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:57.923578 12103 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Bootstrap starting.
I20260620 14:32:57.924782 12103 tablet_bootstrap.cc:654] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:57.925094 12103 log.cc:824] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:57.925470 12102 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Bootstrap starting.
I20260620 14:32:57.925689 12101 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Bootstrap starting.
I20260620 14:32:57.926450 12102 tablet_bootstrap.cc:654] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:57.926754 12102 log.cc:824] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:57.927156 12101 tablet_bootstrap.cc:654] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:57.927495 12102 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: No bootstrap required, opened a new log
I20260620 14:32:57.927496 12101 log.cc:824] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:57.927585 12102 ts_tablet_manager.cc:1397] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:32:57.929572 12102 raft_consensus.cc:348] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.929749 12102 raft_consensus.cc:374] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:57.929783 12102 raft_consensus.cc:729] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d24959c1e5a45bfbba43ba5c7dd55b7, State: Initialized, Role: FOLLOWER
I20260620 14:32:57.929901 12102 consensus_queue.cc:260] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.930174 12102 ts_tablet_manager.cc:1428] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:32:57.930651 12103 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: No bootstrap required, opened a new log
I20260620 14:32:57.930724 12103 ts_tablet_manager.cc:1397] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:32:57.933216 12103 raft_consensus.cc:348] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.933514 12083 heartbeater.cc:503] Master 127.0.98.62:42571 was elected leader, sending a full tablet report...
I20260620 14:32:57.933388 12103 raft_consensus.cc:374] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:57.933815 12103 raft_consensus.cc:729] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4223c8f4df743e4860fd0c6a891a1e1, State: Initialized, Role: FOLLOWER
I20260620 14:32:57.933965 12103 consensus_queue.cc:260] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.934221 12103 ts_tablet_manager.cc:1428] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:32:57.934326 11811 heartbeater.cc:503] Master 127.0.98.62:42571 was elected leader, sending a full tablet report...
I20260620 14:32:57.935547 12101 tablet_bootstrap.cc:492] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: No bootstrap required, opened a new log
I20260620 14:32:57.935618 12101 ts_tablet_manager.cc:1397] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 14:32:57.938047 12101 raft_consensus.cc:348] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.938225 12101 raft_consensus.cc:374] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:57.938277 12101 raft_consensus.cc:729] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e323e06bb77142bbbae984e9987a8c91, State: Initialized, Role: FOLLOWER
I20260620 14:32:57.938395 12101 consensus_queue.cc:260] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:57.938686 12101 ts_tablet_manager.cc:1428] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:32:57.938824 11949 heartbeater.cc:503] Master 127.0.98.62:42571 was elected leader, sending a full tablet report...
Jun 20 14:32:58 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 14:32:58.087031 11812 tablet.cc:2369] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:32:58.109561 11950 tablet.cc:2369] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:32:58.122926 12084 tablet.cc:2369] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:32:58.225901 12107 raft_consensus.cc:482] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:32:58.226059 12107 raft_consensus.cc:504] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:58.226511 12107 leader_election.cc:288] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e323e06bb77142bbbae984e9987a8c91 (127.0.98.2:34017), f4223c8f4df743e4860fd0c6a891a1e1 (127.0.98.1:45467)
I20260620 14:32:58.230450 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.226822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45467 (local address 127.0.98.3:41493)
0620 14:32:58.227021 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:58.227032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:58.227112 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:58.227283 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:58.227284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:58.227286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:58.227368 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:58.227370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.228320 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.228321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:58.229604 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:58.229607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.230184 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.230185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:58.230253 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.230258 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:32:58.230321 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:32:58.230362 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":6,"thread_start_us":97,"threads_started":1}
I20260620 14:32:58.230435 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.226941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41493 (local address 127.0.98.1:45467)
0620 14:32:58.226995 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:58.226998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:58.227095 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:32:58.227195 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:58.227196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:58.227204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:58.227260 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:58.227431 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.228204 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.229687 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.230092 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.230183 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.230194 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:58.230327 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:32:58.230365 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:32:58.230890 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.226807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59027 (local address 127.0.98.2:34017)
0620 14:32:58.226898 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:58.226900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:58.227097 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:32:58.227131 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:58.227132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:58.227146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:58.227193 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:58.227449 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.228204 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.229725 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.230306 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.230379 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.230389 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:58.230787 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:32:58.230828 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:32:58.231192 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f4ef37c8643c484d07fc7f49a87d6" candidate_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" is_pre_election: true
I20260620 14:32:58.231365 11765 raft_consensus.cc:2454] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d24959c1e5a45bfbba43ba5c7dd55b7 in term 0.
I20260620 14:32:58.230679 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.226723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34017 (local address 127.0.98.3:59027)
0620 14:32:58.226961 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:58.226985 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:58.227039 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:58.227312 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:58.227313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:58.227315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:58.227373 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:58.227375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.228297 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.228298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:58.229583 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:58.229586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.230422 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.230423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:58.230516 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.230519 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:32:58.230572 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:32:58.230613 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260620 14:32:58.231592 11971 leader_election.cc:302] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d24959c1e5a45bfbba43ba5c7dd55b7, f4223c8f4df743e4860fd0c6a891a1e1; no voters:
I20260620 14:32:58.231735 12107 raft_consensus.cc:2781] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:32:58.231798 12107 raft_consensus.cc:482] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:32:58.231820 12107 raft_consensus.cc:3037] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:58.231851 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f4ef37c8643c484d07fc7f49a87d6" candidate_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e323e06bb77142bbbae984e9987a8c91" is_pre_election: true
I20260620 14:32:58.231997 11903 raft_consensus.cc:2454] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d24959c1e5a45bfbba43ba5c7dd55b7 in term 0.
I20260620 14:32:58.232741 12107 raft_consensus.cc:504] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:58.232870 12107 leader_election.cc:288] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [CANDIDATE]: Term 1 election: Requested vote from peers e323e06bb77142bbbae984e9987a8c91 (127.0.98.2:34017), f4223c8f4df743e4860fd0c6a891a1e1 (127.0.98.1:45467)
I20260620 14:32:58.233126 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f4ef37c8643c484d07fc7f49a87d6" candidate_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e323e06bb77142bbbae984e9987a8c91"
I20260620 14:32:58.233223 11903 raft_consensus.cc:3037] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:58.233458 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f4ef37c8643c484d07fc7f49a87d6" candidate_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4223c8f4df743e4860fd0c6a891a1e1"
I20260620 14:32:58.233537 11765 raft_consensus.cc:3037] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:58.234092 11903 raft_consensus.cc:2454] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d24959c1e5a45bfbba43ba5c7dd55b7 in term 1.
I20260620 14:32:58.234264 11971 leader_election.cc:302] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d24959c1e5a45bfbba43ba5c7dd55b7, e323e06bb77142bbbae984e9987a8c91; no voters:
I20260620 14:32:58.234341 12107 raft_consensus.cc:2781] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:58.234352 11765 raft_consensus.cc:2454] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d24959c1e5a45bfbba43ba5c7dd55b7 in term 1.
I20260620 14:32:58.234479 12107 raft_consensus.cc:686] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3d24959c1e5a45bfbba43ba5c7dd55b7, State: Running, Role: LEADER
I20260620 14:32:58.234575 12107 consensus_queue.cc:237] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } }
I20260620 14:32:58.235427 11598 catalog_manager.cc:5495] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d24959c1e5a45bfbba43ba5c7dd55b7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d24959c1e5a45bfbba43ba5c7dd55b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43983 } health_report { overall_health: HEALTHY } } }
I20260620 14:32:58.267714 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.264418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.0.98.3:43983)
0620 14:32:58.264461 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:58.264464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:58.264548 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:32:58.264601 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:58.264602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:58.264625 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:58.264717 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:32:58.264932 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.265655 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.266766 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.267063 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.267134 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.267335 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:32:58.267469 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:32:58.267557 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:58.267620 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:32:58.267640 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:32:58.270598 11903 raft_consensus.cc:1261] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 1 FOLLOWER]: Refusing update from remote peer 3d24959c1e5a45bfbba43ba5c7dd55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:58.270598 11765 raft_consensus.cc:1261] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 1 FOLLOWER]: Refusing update from remote peer 3d24959c1e5a45bfbba43ba5c7dd55b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:58.271060 12107 consensus_queue.cc:1035] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e323e06bb77142bbbae984e9987a8c91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:58.271243 12107 consensus_queue.cc:1035] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4223c8f4df743e4860fd0c6a891a1e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:58.274005 12115 mvcc.cc:205] Tried to move back new op lower bound from 7298932646992875520 to 7298932646849261568. Current Snapshot: MvccSnapshot[applied={T|T < 7298932646992875520}]
I20260620 14:32:58.274622 12116 mvcc.cc:205] Tried to move back new op lower bound from 7298932646992875520 to 7298932646849261568. Current Snapshot: MvccSnapshot[applied={T|T < 7298932646992875520}]
I20260620 14:32:58.276309 12117 mvcc.cc:205] Tried to move back new op lower bound from 7298932646992875520 to 7298932646849261568. Current Snapshot: MvccSnapshot[applied={T|T < 7298932646992875520}]
Jun 20 14:32:58 dist-test-slave-tc2g krb5kdc[11123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:32:58.291167 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:58.291272 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:58.292148 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04f4f494124c4cfea437a9725b7c880d: Sending DeleteTablet for 3 replicas of tablet e48f4ef37c8643c484d07fc7f49a87d6
I20260620 14:32:58.292565 11737 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36512
I20260620 14:32:58.292559 11879 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39266
I20260620 14:32:58.292784 12124 tablet_replica.cc:330] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1: stopping tablet replica
I20260620 14:32:58.293017 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f4ef37c8643c484d07fc7f49a87d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48980
I20260620 14:32:58.293066 12124 raft_consensus.cc:2229] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:32:58.293336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 11680
I20260620 14:32:58.293274 12124 raft_consensus.cc:2258] T e48f4ef37c8643c484d07fc7f49a87d6 P f4223c8f4df743e4860fd0c6a891a1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:58.294064 12125 tablet_replica.cc:330] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: stopping tablet replica
I20260620 14:32:58.294245 12125 raft_consensus.cc:2229] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:32:58.294386 12125 raft_consensus.cc:2258] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:58.294904 12125 ts_tablet_manager.cc:1905] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:32:58.296473 12126 tablet_replica.cc:330] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: stopping tablet replica
I20260620 14:32:58.296617 12125 ts_tablet_manager.cc:1918] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:32:58.296721 12125 log.cc:1196] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/e48f4ef37c8643c484d07fc7f49a87d6
I20260620 14:32:58.296998 12125 ts_tablet_manager.cc:1939] T e48f4ef37c8643c484d07fc7f49a87d6 P e323e06bb77142bbbae984e9987a8c91: Deleting consensus metadata
I20260620 14:32:58.297501 11584 catalog_manager.cc:4841] TS e323e06bb77142bbbae984e9987a8c91 (127.0.98.2:34017): tablet e48f4ef37c8643c484d07fc7f49a87d6 (table test-table [id=9044e3d1930a4ac29ca09d760defc834]) successfully deleted
I20260620 14:32:58.297710 12126 raft_consensus.cc:2229] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:32:58.297880 12126 raft_consensus.cc:2258] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:58.298316 12126 ts_tablet_manager.cc:1905] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:32:58.298682 11584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45467 (error 108)
I20260620 14:32:58.298890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 11817
I20260620 14:32:58.306272 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.306149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45467 (local address 127.0.0.1:36526)
0620 14:32:58.306201 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:58.306233 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:32:58.306510 11584 catalog_manager.cc:4584] TS f4223c8f4df743e4860fd0c6a891a1e1 (127.0.98.1:45467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e48f4ef37c8643c484d07fc7f49a87d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45467: connect: Connection refused (error 111)
I20260620 14:32:58.306643 12126 ts_tablet_manager.cc:1918] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:32:58.306761 12126 log.cc:1196] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/e48f4ef37c8643c484d07fc7f49a87d6
I20260620 14:32:58.307080 12126 ts_tablet_manager.cc:1939] T e48f4ef37c8643c484d07fc7f49a87d6 P 3d24959c1e5a45bfbba43ba5c7dd55b7: Deleting consensus metadata
I20260620 14:32:58.307943 11586 catalog_manager.cc:4841] TS 3d24959c1e5a45bfbba43ba5c7dd55b7 (127.0.98.3:43983): tablet e48f4ef37c8643c484d07fc7f49a87d6 (table test-table [id=9044e3d1930a4ac29ca09d760defc834]) successfully deleted
I20260620 14:32:58.308185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 11954
I20260620 14:32:58.314815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 11567
I20260620 14:32:58.322495 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:32:58.667068 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:32:58.667177 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:32:59.012681 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:32:59.012 UTC [11153] LOG: received smart shutdown request
2026-06-20 14:32:59.016 UTC [11153] LOG: background worker "logical replication launcher" (PID 11162) exited with exit code 1
2026-06-20 14:32:59.016 UTC [11157] LOG: shutting down
2026-06-20 14:32:59.162 UTC [11153] LOG: database system is shut down
2026-06-20T14:32:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45618 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:32:59 dist-test-slave-tc2g krb5kdc[12137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:32:59 dist-test-slave-tc2g krb5kdc[12137](info): set up 2 sockets
Jun 20 14:32:59 dist-test-slave-tc2g krb5kdc[12137](info): commencing operation
krb5kdc: starting...
W20260620 14:33:00.309333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:33:00 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:33:00.467185 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpSwtz3/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:33:02.789 UTC [12167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:33:02.790 UTC [12167] LOG: listening on IPv4 address "127.0.98.20", port 32919
2026-06-20 14:33:02.793 UTC [12167] LOG: listening on Unix socket "/tmp/.s.PGSQL.32919"
2026-06-20 14:33:02.813 UTC [12170] LOG: database system was shut down at 2026-06-20 14:33:01 UTC
2026-06-20 14:33:02.816 UTC [12167] LOG: database system is ready to accept connections
I20260620 14:33:03.773253 392 mini_postgres.cc:96] Postgres bound to 32919
2026-06-20 14:33:03.779 UTC [12178] FATAL: database "slave" does not exist
127.0.98.20:32919 - accepting connections
I20260620 14:33:03.779963 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:33:03.794231 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:33:04.238343 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:33:04.238427 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin
2026-06-20 14:33:04,505 [I] DB FLAVOR :POSTGRES
2026-06-20 14:33:04,505 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:33:04,505 [I] Checking connection..
2026-06-20 14:33:04,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:33:04,767 [I] Checking connection passed.
2026-06-20 14:33:04,767 [I] --------- Verifying version history table ---------
2026-06-20 14:33:04,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:33:04,997 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:33:04,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:33:05,258 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:33:05,258 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:33:05,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:33:05.519 UTC [12242] WARNING: there is no transaction in progress
2026-06-20 14:33:05,526 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:33:05,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:33:05,767 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:33:05,768 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:33:05,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:33:06,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:33:06,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tc2g.c.gcp-upstream.internal','N') ;"
2026-06-20 14:33:06,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:33:06,725 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:33:06,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:33:06,971 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:33:06,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:33:07,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:33:07,456 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:33:07,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:33:08.437 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.454 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.471 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.485 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.501 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.518 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.575 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.581 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.595 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.605 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.615 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.628 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.637 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.643 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.649 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:08.655 UTC [12350] WARNING: there is no transaction in progress
2026-06-20 14:33:09,256 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:33:09,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:33:09,490 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:33:09,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:33:09,740 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:33:09,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:33:09,984 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:33:09,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:33:10,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:33:10,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tc2g.c.gcp-upstream.internal';"
2026-06-20 14:33:10,684 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:33:10,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:33:10,940 [I] DB_PATCHES have already been applied
I20260620 14:33:10.945686 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpSwtz3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:33:10.945748 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:33:10.948688 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38741
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:33:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:33:12.051707 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.272s user 0.000s sys 0.008s
Jun 20, 2026 2:33:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:33:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 2:33:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:33:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:33:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:33:33.754823 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:33:33 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966013, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:33:33.833482 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:33:33.842857 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:33:34.665735 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:33:34.665820 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms
I20260620 14:33:34.665838 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:32919
2026-06-20 14:33:34,828 [I] DB FLAVOR :POSTGRES
2026-06-20 14:33:34,829 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:33:34,829 [I] Checking connection
2026-06-20 14:33:34,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:33:35,057 [I] connection success
2026-06-20 14:33:35,057 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:33:35,057 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:33:35,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:33:35,291 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:33:35,291 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:33:35,291 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:33:35,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:33:35,557 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:33:36.032707 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:33:36.244750 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:33:36.286012 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:33:36 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966013, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:36.685495 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:33:36.685709 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpSwtz3/build/release/bin/postgresql.jar:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpSwtz3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:33:36.685730 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:33:36.689566 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12530
I20260620 14:33:36.689639 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56985
Jun 20, 2026 2:33:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:33:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:33:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:33:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:33:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:33:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:33:39.030970 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.197s user 0.000s sys 0.146s
I20260620 14:33:39.031078 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:33:39.031127 392 mini_ranger_kms.cc:401] 127.0.98.20:56985/kms/v1/keys
Jun 20 14:33:39 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:33:39 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:33:39 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:33:42 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:33:42.140411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42051
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:42.225297 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:42.225581 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:42.225638 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:42.226827 12581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:42.226900 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:42.226923 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:42.226945 12581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:42.226974 12581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:42.228591 12581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42051
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:42.228880 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:42.229115 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:42.234560 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:42.234573 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:42.234573 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:42.234602 12581 server_base.cc:1034] running on GCE node
I20260620 14:33:42.235078 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:42.235404 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:42.236562 12581 hybrid_clock.cc:648] HybridClock initialized: now 1781966022236555 us; error 39 us; skew 500 ppm
Jun 20 14:33:42 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:42.239151 12581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:42.239606 12581 webserver.cc:460] Webserver started at http://127.0.98.62:45725/ using document root <none> and password file <none>
I20260620 14:33:42.239837 12581 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:42.239899 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:42.240057 12581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:33:42 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:42.286254 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf"
format_stamp: "Formatted at 2026-06-20 14:33:42 on dist-test-slave-tc2g"
server_key: "e402cbb9034b7aa05254f4c02aac0192"
server_key_iv: "175267cf865d7a89fcca24859319ef0a"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:42.286755 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf"
format_stamp: "Formatted at 2026-06-20 14:33:42 on dist-test-slave-tc2g"
server_key: "e402cbb9034b7aa05254f4c02aac0192"
server_key_iv: "175267cf865d7a89fcca24859319ef0a"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:42.288559 12581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:33:42.302845 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:42.303073 12581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:42.303160 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf"
format_stamp: "Formatted at 2026-06-20 14:33:42 on dist-test-slave-tc2g"
server_key: "e402cbb9034b7aa05254f4c02aac0192"
server_key_iv: "175267cf865d7a89fcca24859319ef0a"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:42.303262 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:42.325093 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:42.327307 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:42.327461 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:42.332458 12581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42051
I20260620 14:33:42.332485 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42051 every 8 connection(s)
I20260620 14:33:42.332898 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:42.333667 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:42.335896 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Bootstrap starting.
I20260620 14:33:42.336408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 12581
I20260620 14:33:42.336530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:33:42.336575 12649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:42.336773 392 external_mini_cluster.cc:1428] Setting key ce28e1932961508a787edeea00862bb8
I20260620 14:33:42.336961 12649 log.cc:824] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:42.337729 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: No bootstrap required, opened a new log
I20260620 14:33:42.340103 12649 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:42.340270 12649 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:42.340299 12649 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca79e3d213e4c5f84aedbc49ebc0ebf, State: Initialized, Role: FOLLOWER
I20260620 14:33:42.340421 12649 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:42.340477 12649 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:42.340502 12649 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:42.340538 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:42.341073 12649 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:42.341154 12649 leader_election.cc:302] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca79e3d213e4c5f84aedbc49ebc0ebf; no voters:
I20260620 14:33:42.341295 12649 leader_election.cc:288] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:42.341323 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:42.341446 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 LEADER]: Becoming Leader. State: Replica: eca79e3d213e4c5f84aedbc49ebc0ebf, State: Running, Role: LEADER
I20260620 14:33:42.341519 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:42.341585 12649 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:42.341823 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260620 14:33:42.341878 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:42.342000 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca79e3d213e4c5f84aedbc49ebc0ebf. Latest consensus state: current_term: 1 leader_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260620 14:33:42.342053 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:42.342206 12658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:42.342700 12658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:42.343005 12649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 14:33:42 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:42.346560 12671 mvcc.cc:205] Tried to move back new op lower bound from 7298932827521282048 to 7298932827511140352. Current Snapshot: MvccSnapshot[applied={T|T < 7298932827521282048}]
I20260620 14:33:42.350492 12658 catalog_manager.cc:1347] Generated new cluster ID: 39943a563ab8417d8c04964ff6048518
I20260620 14:33:42.350545 12658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:42.352445 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:42.337278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.0.98.62:42051)
0620 14:33:42.337456 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:42.337466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:42.337502 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:33:42.338119 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:42.338128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:42.338190 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:42.338347 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:42.339084 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:42.340569 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:42.341729 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:42.342389 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:42.342471 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:42.346324 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:42.346371 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:42.346371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:42.346468 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:42.349731 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:42.350858 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:42.350861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:42.350862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:42.350964 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:42.351192 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:42.351193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:42.351194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:42.351570 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:42.351742 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:42.352010 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:33:42.352179 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260620 14:33:42.361374 12658 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:42.362192 12658 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:42.367136 12658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Generated new TSK 0
I20260620 14:33:42.367362 12658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:33:43.008179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:43.103464 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:43.103698 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:43.103739 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:43.104954 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:43.105021 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:43.105067 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:33:43.106596 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:43.106879 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:43.107164 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:43.112211 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.112232 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.112392 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.113410 12694 server_base.cc:1034] running on GCE node
I20260620 14:33:43.113667 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:43.113924 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:43.117187 12694 hybrid_clock.cc:648] HybridClock initialized: now 1781966023117156 us; error 50 us; skew 500 ppm
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.120492 12694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:33:43.121032 12694 webserver.cc:460] Webserver started at http://127.0.98.1:35215/ using document root <none> and password file <none>
I20260620 14:33:43.121330 12694 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:43.121407 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:43.121582 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:43.136015 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbb72012a2f44c669051a4420d46daf6"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "eeac66fd34bf1eacd81e893f84d8a81a"
server_key_iv: "4f327e93e1582d702c0218397be7ed62"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.136453 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbb72012a2f44c669051a4420d46daf6"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "eeac66fd34bf1eacd81e893f84d8a81a"
server_key_iv: "4f327e93e1582d702c0218397be7ed62"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.138307 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:33:43.152417 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.152637 12694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:43.152720 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "dbb72012a2f44c669051a4420d46daf6"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "eeac66fd34bf1eacd81e893f84d8a81a"
server_key_iv: "4f327e93e1582d702c0218397be7ed62"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.152817 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:43.171909 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:43.174366 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:43.174520 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:43.174762 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:43.175194 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:43.175251 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.175284 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:43.175334 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.182337 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38385
I20260620 14:33:43.182407 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38385 every 8 connection(s)
I20260620 14:33:43.182799 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:43.184186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 12694
I20260620 14:33:43.184324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:33:43.184525 392 external_mini_cluster.cc:1428] Setting key c4864cd71e953486f234a315aef28230
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:43.193909 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.183912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60155 (local address 127.0.98.62:42051)
0620 14:33:43.184796 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:43.184799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:43.184808 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:43.184831 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:43.184831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:43.184844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:43.184905 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:43.185780 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.186582 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.187605 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.187831 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.187917 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.190258 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:43.190260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:43.190260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:43.190268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:43.192215 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.192728 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.192729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.192729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.192809 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.193099 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.193100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.193100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.193236 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:43.193303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:43.193681 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:33:43.193802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":83,"threads_started":1}
I20260620 14:33:43.194026 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.183720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.1:60155)
0620 14:33:43.183940 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:43.183993 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:43.184686 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:43.185002 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:43.185007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:43.185351 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:43.185650 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.185658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.186686 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.186687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.187495 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.187497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.187941 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.187942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.188066 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.188606 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:43.188614 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:43.190152 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:43.192324 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.192331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.192334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.192633 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.192887 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.192888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.192888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.193023 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.193317 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:43.193323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:43.193524 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 14:33:43.193647 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:33:43.193800 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260620 14:33:43.194613 12823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:43.194718 12823 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:43.194912 12823 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:43.195511 12613 ts_manager.cc:194] Registered new tserver with Master: dbb72012a2f44c669051a4420d46daf6 (127.0.98.1:38385)
I20260620 14:33:43.196419 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60155
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:33:43.223690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:43.327770 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:43.328054 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:43.328088 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:43.329831 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:43.329888 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:43.329928 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:33:43.332095 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:43.332338 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:43.332630 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:43.338809 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.338871 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.339010 12831 server_base.cc:1034] running on GCE node
W20260620 14:33:43.339075 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.339497 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:43.339790 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:43.340970 12831 hybrid_clock.cc:648] HybridClock initialized: now 1781966023340952 us; error 34 us; skew 500 ppm
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.344343 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:33:43.344899 12831 webserver.cc:460] Webserver started at http://127.0.98.2:46683/ using document root <none> and password file <none>
I20260620 14:33:43.345158 12831 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:43.345220 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:43.345393 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:43.359012 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "916e53ec1a944383a635d1e75f3dea62"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "6dd61972265a047710cac04248e03b10"
server_key_iv: "66629b9bb56b918c2a0a61901127ea31"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.359515 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "916e53ec1a944383a635d1e75f3dea62"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "6dd61972265a047710cac04248e03b10"
server_key_iv: "66629b9bb56b918c2a0a61901127ea31"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.361284 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.378118 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.378520 12831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:43.378652 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "916e53ec1a944383a635d1e75f3dea62"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "6dd61972265a047710cac04248e03b10"
server_key_iv: "66629b9bb56b918c2a0a61901127ea31"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.378767 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:43.394332 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:43.396785 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:43.396986 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:43.397272 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:43.397720 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:43.397816 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.397895 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:43.397960 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.404284 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36291
I20260620 14:33:43.404834 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:43.407853 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36291 every 8 connection(s)
I20260620 14:33:43.409443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 12831
I20260620 14:33:43.409554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:33:43.409762 392 external_mini_cluster.cc:1428] Setting key 47fc33580c702e5d3ae0ea6862ca113a
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:43.422242 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.408661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48303 (local address 127.0.98.62:42051)
0620 14:33:43.408927 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:43.408930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:43.408984 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:33:43.409972 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:43.409973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:43.409985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:43.410021 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:43.410914 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.411710 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.413234 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.413563 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.413623 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.418628 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:43.418630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:43.418630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:43.418638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:43.420623 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.421111 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.421112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.421113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.421169 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.421463 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.421463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.421464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.421617 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:43.421662 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:43.422043 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 14:33:43.422139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260620 14:33:43.423220 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.408640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.2:48303)
0620 14:33:43.408836 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:43.408889 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:43.409788 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:43.410218 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:43.410221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:43.410477 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:43.410802 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.410810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.411831 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.411832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.413132 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.413135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.413753 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.413754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.413900 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.416746 (+ 2846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:43.416758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:43.418490 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:43.420724 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.420731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.420734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.421022 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.421240 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.421240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.421240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.421366 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.421710 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:43.421716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:43.421895 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:33:43.422591 (+ 696us) client_negotiation.cc:240] Negotiation successful
0620 14:33:43.422775 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260620 14:33:43.423935 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:43.424036 12962 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:43.424218 12962 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:43.424605 12613 ts_manager.cc:194] Registered new tserver with Master: 916e53ec1a944383a635d1e75f3dea62 (127.0.98.2:36291)
I20260620 14:33:43.425122 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48303
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:33:43.446339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:43.570505 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:43.570794 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:43.570843 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:43.572635 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:43.572702 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:43.572746 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:33:43.574991 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:43.575255 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:43.575573 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:43.581873 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.582054 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.582293 12967 server_base.cc:1034] running on GCE node
W20260620 14:33:43.582355 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.582603 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:43.582921 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:43.584085 12967 hybrid_clock.cc:648] HybridClock initialized: now 1781966023584058 us; error 48 us; skew 500 ppm
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.587399 12967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:33:43.587951 12967 webserver.cc:460] Webserver started at http://127.0.98.3:43675/ using document root <none> and password file <none>
I20260620 14:33:43.588227 12967 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:43.588305 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:43.588445 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:43.604877 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ea2f9fe59274a578dcf0ba216b7cc61"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "680eb3d515e5d7ac68e59fa7e2e616cf"
server_key_iv: "f0045859dab5b73f4ce5433b834e16a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.605286 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ea2f9fe59274a578dcf0ba216b7cc61"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "680eb3d515e5d7ac68e59fa7e2e616cf"
server_key_iv: "f0045859dab5b73f4ce5433b834e16a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.607349 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:33:43.630336 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.630559 12967 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260620 14:33:43.630666 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "5ea2f9fe59274a578dcf0ba216b7cc61"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "680eb3d515e5d7ac68e59fa7e2e616cf"
server_key_iv: "f0045859dab5b73f4ce5433b834e16a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.630780 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:43.641687 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:43.643924 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:43.644085 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:43.644325 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:43.644694 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:43.644749 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.644793 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:43.644819 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.652114 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40223
I20260620 14:33:43.652205 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40223 every 8 connection(s)
I20260620 14:33:43.652530 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:43.653020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 12967
I20260620 14:33:43.653146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:33:43.653357 392 external_mini_cluster.cc:1428] Setting key 422499ff3fcffd8642cfb58dc8cc3ce5
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:43.662901 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.653287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59215 (local address 127.0.98.62:42051)
0620 14:33:43.653349 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:43.653351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:43.653756 (+ 405us) server_negotiation.cc:407] Connection header received
0620 14:33:43.654454 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:43.654455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:43.654466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:43.654511 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:43.655084 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.655588 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.656586 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.656837 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.656908 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.659234 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:43.659235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:43.659236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:43.659244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:43.660933 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.661478 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.661479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.661480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.661539 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:43.661859 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:43.661860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:43.661860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:43.662127 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:43.662202 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:43.662710 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 14:33:43.662800 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:33:43.663160 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:43.653349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.3:59215)
0620 14:33:43.653557 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:43.653686 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:43.654349 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:43.654582 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:43.654586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:43.654774 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:43.654982 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.654986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.655680 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.655681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.656479 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:43.656481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:43.656930 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:43.656931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:43.657062 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:43.657626 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:43.657634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:43.659135 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:43.661026 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.661035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.661038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.661363 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.661611 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:43.661613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:43.661613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:43.661770 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:43.662231 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:43.662238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:43.662515 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 14:33:43.662653 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:33:43.662866 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
I20260620 14:33:43.663641 13097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:43.663734 13097 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:43.663941 13097 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:43.664343 12613 ts_manager.cc:194] Registered new tserver with Master: 5ea2f9fe59274a578dcf0ba216b7cc61 (127.0.98.3:40223)
I20260620 14:33:43.664705 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59215
I20260620 14:33:43.665870 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:43.666438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 12694
I20260620 14:33:43.672945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 12831
I20260620 14:33:43.681622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 12967
I20260620 14:33:43.686345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 12581
I20260620 14:33:43.693496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--webserver_interface=127.0.98.62
--webserver_port=45725
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42051
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:43.774210 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:43.774420 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:43.774442 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:43.776104 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:43.776149 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:43.776163 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:43.776176 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:43.776188 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:43.777769 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42051
--ranger_config_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:43.777945 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:43.778103 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:43.782846 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.782822 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.782831 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.782893 13100 server_base.cc:1034] running on GCE node
I20260620 14:33:43.783456 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:43.783756 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:43.784922 13100 hybrid_clock.cc:648] HybridClock initialized: now 1781966023784915 us; error 34 us; skew 500 ppm
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.787472 13100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:43.787928 13100 webserver.cc:460] Webserver started at http://127.0.98.62:45725/ using document root <none> and password file <none>
I20260620 14:33:43.788102 13100 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:43.788152 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:43.835378 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:43.836125 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:43.836329 13100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:43.836417 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf"
format_stamp: "Formatted at 2026-06-20 14:33:42 on dist-test-slave-tc2g"
server_key: "e402cbb9034b7aa05254f4c02aac0192"
server_key_iv: "175267cf865d7a89fcca24859319ef0a"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:43.836788 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:43.848835 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:43.850605 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:43.850775 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:43.855113 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42051
I20260620 14:33:43.855149 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42051 every 8 connection(s)
I20260620 14:33:43.855537 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:43.856709 13169 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:43.857389 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Bootstrap starting.
I20260620 14:33:43.858680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13100
I20260620 14:33:43.858963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38385
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42051
--webserver_port=35215
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:43.865612 13169 log.cc:824] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:43.867341 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:43.867636 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Bootstrap complete.
I20260620 14:33:43.870152 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:43.870414 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca79e3d213e4c5f84aedbc49ebc0ebf, State: Initialized, Role: FOLLOWER
I20260620 14:33:43.870611 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:43.870692 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:43.870745 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:43.870795 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:33:43.871821 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:43.871953 13169 leader_election.cc:302] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca79e3d213e4c5f84aedbc49ebc0ebf; no voters:
I20260620 14:33:43.872140 13169 leader_election.cc:288] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:33:43.872190 13173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:33:43.872330 13173 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [term 2 LEADER]: Becoming Leader. State: Replica: eca79e3d213e4c5f84aedbc49ebc0ebf, State: Running, Role: LEADER
I20260620 14:33:43.872444 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:43.872432 13173 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260620 14:33:43.872725 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca79e3d213e4c5f84aedbc49ebc0ebf. Latest consensus state: current_term: 2 leader_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260620 14:33:43.872814 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:43.873216 13177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:43.873761 13177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:43.873871 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca79e3d213e4c5f84aedbc49ebc0ebf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260620 14:33:43.873929 13177 catalog_manager.cc:1259] Loaded cluster ID: 39943a563ab8417d8c04964ff6048518
I20260620 14:33:43.874025 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:43.874045 13177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:43.874349 13169 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:33:43.877549 13177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:43.877934 13177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Loaded TSK: 0
I20260620 14:33:43.878163 13177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:33:43.965714 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:43.965978 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:43.966017 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:43.967413 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:43.967490 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:43.967537 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:33:43.969017 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35215
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:43.969285 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:43.969559 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:43.974603 13206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.974573 13207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:43.974928 13209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:43.975162 13171 server_base.cc:1034] running on GCE node
I20260620 14:33:43.975355 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:43.975600 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:43.976758 13171 hybrid_clock.cc:648] HybridClock initialized: now 1781966023976735 us; error 37 us; skew 500 ppm
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:43.980083 13171 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:33:43.980619 13171 webserver.cc:460] Webserver started at http://127.0.98.1:35215/ using document root <none> and password file <none>
I20260620 14:33:43.980871 13171 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:43.980944 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:43 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:44.000852 13171 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:33:44.001825 13217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.002023 13171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:44.002105 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "dbb72012a2f44c669051a4420d46daf6"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "eeac66fd34bf1eacd81e893f84d8a81a"
server_key_iv: "4f327e93e1582d702c0218397be7ed62"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:44.002526 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:44.014506 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:44.017143 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:44.017313 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:44.017567 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:44.017959 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:44.018011 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.018049 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:44.018074 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.025238 13171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38385
I20260620 14:33:44.025764 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:44.025764 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38385 every 8 connection(s)
I20260620 14:33:44.034934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13171
I20260620 14:33:44.035255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36291
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42051
--webserver_port=46683
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:44.042066 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.026616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.1:54691)
0620 14:33:44.026961 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:44.027007 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:44.029735 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:44.030395 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:44.030399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:44.030601 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:44.030791 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.030796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.031953 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.031954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.033177 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.033180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.034062 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.034063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.034199 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.034824 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:44.034835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:44.037705 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:44.040108 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.040115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.040117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.040422 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.040741 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.040742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.040742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.040877 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.041361 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:44.041366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:44.041552 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 14:33:44.041658 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:33:44.041819 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":89,"threads_started":1}
I20260620 14:33:44.042789 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.026977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54691 (local address 127.0.98.62:42051)
0620 14:33:44.029957 (+ 2980us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:44.029967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:44.030005 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:33:44.030161 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:44.030166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:44.030212 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:44.030355 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:44.030905 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.031831 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.033280 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.033940 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.034485 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.037836 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:44.037841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:44.037841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:44.037862 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:44.040000 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.040520 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.040524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.040525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.040650 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.040962 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.040963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.040964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.041269 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:44.042378 (+ 1109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:44.042453 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:33:44.042577 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2922,"thread_start_us":78,"threads_started":1}
I20260620 14:33:44.043509 13331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:44.043629 13331 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.043883 13331 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.044726 13331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:33:44.044811 13331 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.044878 13331 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:44.045179 13331 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.045259 13331 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.045550 13331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:44.160050 13335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:44.160382 13335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:44.160430 13335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:44.162326 13335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:44.162401 13335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:44.162447 13335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:33:44.164846 13335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46683
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:44.165280 13335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:44.165652 13335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:44.172574 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:44.173139 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:44.173439 13343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:44.179524 13335 server_base.cc:1034] running on GCE node
I20260620 14:33:44.179832 13335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:44.180172 13335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:44.181336 13335 hybrid_clock.cc:648] HybridClock initialized: now 1781966024181329 us; error 34 us; skew 500 ppm
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:44.184077 13335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:33:44.184610 13335 webserver.cc:460] Webserver started at http://127.0.98.2:46683/ using document root <none> and password file <none>
I20260620 14:33:44.184871 13335 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:44.184928 13335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:44.206986 13335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:44.207769 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.207999 13335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:44.208094 13335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "916e53ec1a944383a635d1e75f3dea62"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "6dd61972265a047710cac04248e03b10"
server_key_iv: "66629b9bb56b918c2a0a61901127ea31"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:44.208474 13335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:44.245132 13335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:44.247795 13335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:44.247959 13335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:44.248250 13335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:44.248621 13335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:44.248694 13335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.248749 13335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:44.248776 13335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.256386 13335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36291
I20260620 14:33:44.256433 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36291 every 8 connection(s)
I20260620 14:33:44.256821 13335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:44.264616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13335
I20260620 14:33:44.264935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40223
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42051
--webserver_port=43675
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:44.266717 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.257915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53297 (local address 127.0.98.62:42051)
0620 14:33:44.257973 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:44.257975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:44.257990 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:33:44.258491 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:44.258493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:44.258510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:44.258557 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:44.259175 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.259693 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.260706 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.260934 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.261019 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.263086 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:44.263088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:44.263089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:44.263097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:44.264700 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.265273 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.265275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.265276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.265346 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.265682 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.265683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.265683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.265842 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:44.265911 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:44.266440 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:33:44.266554 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:33:44.267556 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.257624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.2:53297)
0620 14:33:44.257786 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:44.257830 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:44.258376 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:44.258639 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:44.258643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:44.258844 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:44.259072 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.259078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.259807 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.259808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.260613 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.260615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.261040 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.261041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.261167 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.261756 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:44.261765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:44.262989 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:44.264798 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.264806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.264808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.265148 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.265449 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.265450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.265451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.265603 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.265940 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:44.265948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:44.266138 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 14:33:44.267093 (+ 955us) client_negotiation.cc:240] Negotiation successful
0620 14:33:44.267285 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20260620 14:33:44.268234 13465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:44.268364 13465 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.268597 13465 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.269253 13465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:33:44.269374 13465 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.269522 13465 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:44.269901 13465 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.270018 13465 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.270339 13465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:44.363987 13468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:44.364248 13468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:44.364282 13468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:44.365825 13468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:44.365888 13468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:44.365935 13468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:33:44.367588 13468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43675
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:44.367862 13468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:44.368120 13468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:44.372905 13473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:44.372898 13474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:44.373893 13468 server_base.cc:1034] running on GCE node
W20260620 14:33:44.374291 13476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:44.374562 13468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:44.374823 13468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:44.390381 13468 hybrid_clock.cc:648] HybridClock initialized: now 1781966024390360 us; error 41 us; skew 500 ppm
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:44.393277 13468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:33:44.393980 13468 webserver.cc:460] Webserver started at http://127.0.98.3:43675/ using document root <none> and password file <none>
I20260620 14:33:44.394258 13468 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:44.394342 13468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:44.408524 13468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:44.409478 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.409693 13468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:44.409789 13468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "5ea2f9fe59274a578dcf0ba216b7cc61"
format_stamp: "Formatted at 2026-06-20 14:33:43 on dist-test-slave-tc2g"
server_key: "680eb3d515e5d7ac68e59fa7e2e616cf"
server_key_iv: "f0045859dab5b73f4ce5433b834e16a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:44.410285 13468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:44.442042 13468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:44.444691 13468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:44.444859 13468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:44.445143 13468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:44.445562 13468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:44.445622 13468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.445667 13468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:44.445693 13468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:44.453011 13468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40223
I20260620 14:33:44.453112 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40223 every 8 connection(s)
I20260620 14:33:44.453500 13468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:44.461922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13468
I20260620 14:33:44.465325 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.454294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.3:51709)
0620 14:33:44.454469 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:44.454516 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:44.455040 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:44.455307 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:44.455310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:44.455501 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:44.455696 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.455700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.456641 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.456642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.458764 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:44.458767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.459336 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.459337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:44.459476 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.460124 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:44.460134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:44.461600 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:44.463660 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.463667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.463669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.463947 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.464168 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:44.464169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:44.464169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:44.464302 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:44.464608 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:44.464614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:44.464790 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:33:44.464923 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:33:44.465079 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260620 14:33:44.465696 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.454265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51709 (local address 127.0.98.62:42051)
0620 14:33:44.454316 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:44.454318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:44.454626 (+ 308us) server_negotiation.cc:407] Connection header received
0620 14:33:44.455146 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:44.455148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:44.455163 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:44.455252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:44.455792 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.456536 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.458876 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.459163 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.459224 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.461700 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:44.461702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:44.461702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:44.461710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:44.463564 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.464032 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.464033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.464033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.464093 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.464384 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.464385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.464385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.464517 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:44.464581 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:44.465532 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 14:33:44.465607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:33:44.466060 13599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260620 14:33:44.466164 13599 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.466362 13599 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.466955 13599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:33:44.467029 13599 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.467103 13599 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:44.467397 13599 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:44.467476 13599 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
W20260620 14:33:44.467784 13599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:33:44.470881 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:44.462371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.98.62:42051)
0620 14:33:44.462481 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:44.462483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:44.462677 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:33:44.462699 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:44.462700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:44.462712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:44.462772 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:44.463558 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.464302 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.465612 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:44.465912 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:44.465982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:44.467510 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:44.467523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:44.467524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:44.467530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:44.469276 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.469705 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.469706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.469707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.469766 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:44.469963 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:44.469964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:44.469964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:44.470082 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:44.470250 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:44.470699 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 14:33:44.470787 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
Jun 20 14:33:44 dist-test-slave-tc2g krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:45.045763 13331 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:45.045928 13331 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:45.046414 13133 ts_manager.cc:194] Registered new tserver with Master: dbb72012a2f44c669051a4420d46daf6 (127.0.98.1:38385)
I20260620 14:33:45.047465 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54691
I20260620 14:33:45.270565 13465 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:45.270738 13465 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:45.271037 13133 ts_manager.cc:194] Registered new tserver with Master: 916e53ec1a944383a635d1e75f3dea62 (127.0.98.2:36291)
I20260620 14:33:45.271575 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53297
Jun 20 14:33:45 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:45.467995 13599 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:45.468156 13599 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260620 14:33:45.468457 13133 ts_manager.cc:194] Registered new tserver with Master: 5ea2f9fe59274a578dcf0ba216b7cc61 (127.0.98.3:40223)
I20260620 14:33:45.468957 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51709
I20260620 14:33:45.477865 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:45.492479 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.478960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.0.98.62:42051)
0620 14:33:45.480825 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.480828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.480954 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:33:45.481063 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.481065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.481078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.481128 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:45.481998 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.482808 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.483882 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.484236 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.484313 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.489471 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:45.489473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:45.489473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:45.489482 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:45.491200 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:45.491617 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:45.491618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:45.491619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:45.491695 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:45.491915 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:45.491916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:45.491916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:45.492046 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:45.492139 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.492297 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.492383 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":83,"threads_started":1}
I20260620 14:33:45.494802 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:45.496970 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:45.507035 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.500263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.98.2:36291)
0620 14:33:45.501560 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.501564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.501716 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:33:45.501761 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.501764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.501782 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.501874 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:45.502172 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.503678 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.505692 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.506479 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.506557 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.506640 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.506897 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.506943 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":48,"threads_started":1}
I20260620 14:33:45.508991 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.499918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38385 (local address 127.0.0.1:60698)
0620 14:33:45.500063 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.500104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:45.500181 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:45.502764 (+ 2583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:45.502764 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:45.502767 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:45.502816 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.502818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.506757 (+ 3939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.506758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.507811 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.507814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.508710 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.508711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.508780 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.508786 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:33:45.508849 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:33:45.508893 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 14:33:45.509088 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.500454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.0.98.1:38385)
0620 14:33:45.502201 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.502208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.502218 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:45.502254 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.502257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.502273 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.502898 (+ 625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:45.502975 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.504020 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.507921 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.508611 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.508685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.508753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.508961 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.509007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1700,"thread_start_us":49,"threads_started":1}
I20260620 14:33:45.510100 13265 tablet_service.cc:1467] Processing CreateTablet for tablet b33889624f4442bb9d1258929b3422e5 (DEFAULT_TABLE table=test-table [id=78b9319907d440fcbcd42ea13538c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:45.508904 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.500154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36291 (local address 127.0.0.1:46444)
0620 14:33:45.501617 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.501627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:45.501692 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:45.501991 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:45.501995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:45.501998 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:45.502078 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.502080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.503780 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.503781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.505560 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.505563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.506679 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.506680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.506762 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.506775 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:33:45.506848 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:33:45.508832 (+ 1984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"mutex_wait_us":1913,"thread_start_us":64,"threads_started":1}
I20260620 14:33:45.510464 13265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33889624f4442bb9d1258929b3422e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:45.514205 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.500324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40223 (local address 127.0.0.1:37094)
0620 14:33:45.501777 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.501787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:45.501835 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:45.502365 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:45.502366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:45.502368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:45.502414 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.502415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.505113 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.505114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.506425 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.506427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.513877 (+ 7450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.513877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.513945 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.513950 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:33:45.514017 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:33:45.514069 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":56,"threads_started":1}
I20260620 14:33:45.514277 13399 tablet_service.cc:1467] Processing CreateTablet for tablet b33889624f4442bb9d1258929b3422e5 (DEFAULT_TABLE table=test-table [id=78b9319907d440fcbcd42ea13538c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:45.514631 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33889624f4442bb9d1258929b3422e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:45.514650 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.500426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.0.98.3:40223)
0620 14:33:45.500573 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.500578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.501905 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 14:33:45.501952 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.501955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.501974 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.502050 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:45.502505 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.505017 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.506518 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.510100 (+ 3582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.510178 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.510252 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.514496 (+ 4244us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.514548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260620 14:33:45.515355 13529 tablet_service.cc:1467] Processing CreateTablet for tablet b33889624f4442bb9d1258929b3422e5 (DEFAULT_TABLE table=test-table [id=78b9319907d440fcbcd42ea13538c92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:45.515676 13529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33889624f4442bb9d1258929b3422e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:45.517269 13633 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Bootstrap starting.
I20260620 14:33:45.517946 13633 tablet_bootstrap.cc:654] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:45.518323 13633 log.cc:824] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:45.518709 13632 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: Bootstrap starting.
I20260620 14:33:45.519477 13632 tablet_bootstrap.cc:654] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:45.519747 13632 log.cc:824] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:45.520481 13633 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: No bootstrap required, opened a new log
I20260620 14:33:45.520545 13633 ts_tablet_manager.cc:1397] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:45.523031 13633 raft_consensus.cc:348] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.523478 13633 raft_consensus.cc:374] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:45.523533 13633 raft_consensus.cc:729] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916e53ec1a944383a635d1e75f3dea62, State: Initialized, Role: FOLLOWER
I20260620 14:33:45.523659 13633 consensus_queue.cc:260] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.523896 13633 ts_tablet_manager.cc:1428] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:45.524003 13465 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
I20260620 14:33:45.524953 13632 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: No bootstrap required, opened a new log
I20260620 14:33:45.525022 13632 ts_tablet_manager.cc:1397] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 14:33:45.525589 13635 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Bootstrap starting.
I20260620 14:33:45.526276 13635 tablet_bootstrap.cc:654] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:45.526566 13635 log.cc:824] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:45.527942 13632 raft_consensus.cc:348] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.528226 13632 raft_consensus.cc:374] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:45.528319 13632 raft_consensus.cc:729] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb72012a2f44c669051a4420d46daf6, State: Initialized, Role: FOLLOWER
I20260620 14:33:45.529645 13632 consensus_queue.cc:260] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.530144 13632 ts_tablet_manager.cc:1428] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:33:45.530306 13331 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
I20260620 14:33:45.530877 13635 tablet_bootstrap.cc:492] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: No bootstrap required, opened a new log
I20260620 14:33:45.530942 13635 ts_tablet_manager.cc:1397] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 14:33:45.533759 13635 raft_consensus.cc:348] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.533983 13635 raft_consensus.cc:374] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:45.534039 13635 raft_consensus.cc:729] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea2f9fe59274a578dcf0ba216b7cc61, State: Initialized, Role: FOLLOWER
I20260620 14:33:45.534214 13635 consensus_queue.cc:260] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.534526 13635 ts_tablet_manager.cc:1428] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:45.534623 13599 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
Jun 20 14:33:45 dist-test-slave-tc2g krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:33:45.698587 13640 raft_consensus.cc:482] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:45.698745 13640 raft_consensus.cc:504] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.699136 13640 leader_election.cc:288] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb72012a2f44c669051a4420d46daf6 (127.0.98.1:38385), 916e53ec1a944383a635d1e75f3dea62 (127.0.98.2:36291)
I20260620 14:33:45.703466 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.699945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45965 (local address 127.0.98.2:36291)
0620 14:33:45.699992 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.699994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.700004 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:45.700090 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.700092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.700101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.700130 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:45.700342 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.701037 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.702504 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.703080 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.703127 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.703137 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.703353 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.703396 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:33:45.703603 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.699571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52719 (local address 127.0.98.1:38385)
0620 14:33:45.699620 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.699622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.699781 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:33:45.699896 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.699897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.699911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.699944 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:45.700210 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.700927 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.702522 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.703135 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.703186 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.703198 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.703483 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.703529 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:33:45.703715 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.699777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36291 (local address 127.0.98.3:45965)
0620 14:33:45.699948 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.699957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:45.700008 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:45.700205 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:45.700206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:45.700207 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:45.700272 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.700274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.701133 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.701133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.702399 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.702402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.703207 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.703207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.703274 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.703278 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:33:45.703583 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 14:33:45.703621 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
I20260620 14:33:45.703999 13419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33889624f4442bb9d1258929b3422e5" candidate_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916e53ec1a944383a635d1e75f3dea62" is_pre_election: true
I20260620 14:33:45.704174 13419 raft_consensus.cc:2454] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea2f9fe59274a578dcf0ba216b7cc61 in term 0.
I20260620 14:33:45.704262 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.699516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38385 (local address 127.0.98.3:52719)
0620 14:33:45.699693 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.699706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:45.699812 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:45.700024 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:45.700025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:45.700027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:45.700123 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.700125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.701040 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.701041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.702411 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:45.702414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.703302 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.703303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:45.703389 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.703394 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:33:45.704123 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 14:33:45.704171 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":22,"thread_start_us":116,"threads_started":1}
I20260620 14:33:45.704478 13486 leader_election.cc:302] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea2f9fe59274a578dcf0ba216b7cc61, 916e53ec1a944383a635d1e75f3dea62; no voters:
I20260620 14:33:45.704593 13640 raft_consensus.cc:2781] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:45.704561 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33889624f4442bb9d1258929b3422e5" candidate_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb72012a2f44c669051a4420d46daf6" is_pre_election: true
I20260620 14:33:45.704631 13640 raft_consensus.cc:482] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:45.704651 13640 raft_consensus.cc:3037] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:45.704728 13285 raft_consensus.cc:2454] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea2f9fe59274a578dcf0ba216b7cc61 in term 0.
W20260620 14:33:45.704905 13600 tablet.cc:2369] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:45.705572 13640 raft_consensus.cc:504] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.705704 13640 leader_election.cc:288] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [CANDIDATE]: Term 1 election: Requested vote from peers dbb72012a2f44c669051a4420d46daf6 (127.0.98.1:38385), 916e53ec1a944383a635d1e75f3dea62 (127.0.98.2:36291)
I20260620 14:33:45.705839 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33889624f4442bb9d1258929b3422e5" candidate_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb72012a2f44c669051a4420d46daf6"
I20260620 14:33:45.705871 13419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33889624f4442bb9d1258929b3422e5" candidate_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916e53ec1a944383a635d1e75f3dea62"
I20260620 14:33:45.705916 13285 raft_consensus.cc:3037] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:45.705930 13419 raft_consensus.cc:3037] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:45.706772 13419 raft_consensus.cc:2454] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea2f9fe59274a578dcf0ba216b7cc61 in term 1.
I20260620 14:33:45.706905 13285 raft_consensus.cc:2454] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea2f9fe59274a578dcf0ba216b7cc61 in term 1.
I20260620 14:33:45.707042 13486 leader_election.cc:302] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea2f9fe59274a578dcf0ba216b7cc61, 916e53ec1a944383a635d1e75f3dea62; no voters:
I20260620 14:33:45.707353 13640 raft_consensus.cc:2781] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:45.707508 13640 raft_consensus.cc:686] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea2f9fe59274a578dcf0ba216b7cc61, State: Running, Role: LEADER
I20260620 14:33:45.707589 13640 consensus_queue.cc:237] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } }
I20260620 14:33:45.708366 13133 catalog_manager.cc:5495] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea2f9fe59274a578dcf0ba216b7cc61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea2f9fe59274a578dcf0ba216b7cc61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40223 } health_report { overall_health: HEALTHY } } }
I20260620 14:33:45.732405 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.728973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.3:40223)
0620 14:33:45.729079 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:45.729081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:45.729088 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:33:45.729128 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:45.729129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:45.729154 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:45.729221 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:45.729440 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.730164 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.731291 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:45.731593 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:45.731679 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:45.731891 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:45.732028 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:45.732107 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:45.732288 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:33:45.732311 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:33:45.734365 13419 raft_consensus.cc:1261] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 1 FOLLOWER]: Refusing update from remote peer 5ea2f9fe59274a578dcf0ba216b7cc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:45.734768 13640 consensus_queue.cc:1035] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916e53ec1a944383a635d1e75f3dea62" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:45.736701 13285 raft_consensus.cc:1261] T b33889624f4442bb9d1258929b3422e5 P dbb72012a2f44c669051a4420d46daf6 [term 1 FOLLOWER]: Refusing update from remote peer 5ea2f9fe59274a578dcf0ba216b7cc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:45.737030 13640 consensus_queue.cc:1035] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb72012a2f44c669051a4420d46daf6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:45.740036 13651 mvcc.cc:205] Tried to move back new op lower bound from 7298932841404870656 to 7298932841298743296. Current Snapshot: MvccSnapshot[applied={T|T < 7298932841404870656}]
I20260620 14:33:45.742076 13649 mvcc.cc:205] Tried to move back new op lower bound from 7298932841404870656 to 7298932841298743296. Current Snapshot: MvccSnapshot[applied={T|T < 7298932841404870656}]
I20260620 14:33:45.743269 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:45.743386 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:45.744105 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eca79e3d213e4c5f84aedbc49ebc0ebf: Sending DeleteTablet for 3 replicas of tablet b33889624f4442bb9d1258929b3422e5
I20260620 14:33:45.744494 13399 tablet_service.cc:1514] Processing DeleteTablet for tablet b33889624f4442bb9d1258929b3422e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46444
I20260620 14:33:45.744474 13265 tablet_service.cc:1514] Processing DeleteTablet for tablet b33889624f4442bb9d1258929b3422e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60698
I20260620 14:33:45.744695 13529 tablet_service.cc:1514] Processing DeleteTablet for tablet b33889624f4442bb9d1258929b3422e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37094
I20260620 14:33:45.745033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13171
I20260620 14:33:45.756232 13659 tablet_replica.cc:330] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: stopping tablet replica
I20260620 14:33:45.756385 13659 raft_consensus.cc:2229] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:45.756531 13659 raft_consensus.cc:2258] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:45.756721 13658 tablet_replica.cc:330] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: stopping tablet replica
I20260620 14:33:45.756845 13658 raft_consensus.cc:2229] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:45.756982 13658 raft_consensus.cc:2258] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:45.757498 13659 ts_tablet_manager.cc:1905] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:45.757498 13658 ts_tablet_manager.cc:1905] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:33:45.759311 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38385 (error 108)
I20260620 14:33:45.759526 13659 ts_tablet_manager.cc:1918] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:45.759595 13659 log.cc:1196] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/b33889624f4442bb9d1258929b3422e5
I20260620 14:33:45.759776 13658 ts_tablet_manager.cc:1918] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:45.759881 13659 ts_tablet_manager.cc:1939] T b33889624f4442bb9d1258929b3422e5 P 5ea2f9fe59274a578dcf0ba216b7cc61: Deleting consensus metadata
I20260620 14:33:45.759963 13658 log.cc:1196] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/b33889624f4442bb9d1258929b3422e5
I20260620 14:33:45.760277 13658 ts_tablet_manager.cc:1939] T b33889624f4442bb9d1258929b3422e5 P 916e53ec1a944383a635d1e75f3dea62: Deleting consensus metadata
I20260620 14:33:45.760311 13117 catalog_manager.cc:4841] TS 5ea2f9fe59274a578dcf0ba216b7cc61 (127.0.98.3:40223): tablet b33889624f4442bb9d1258929b3422e5 (table test-table [id=78b9319907d440fcbcd42ea13538c92e]) successfully deleted
I20260620 14:33:45.760514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13335
I20260620 14:33:45.766945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13468
I20260620 14:33:45.767434 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.767295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38385 (local address 127.0.0.1:60712)
0620 14:33:45.767357 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.767390 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:33:45.767828 13118 catalog_manager.cc:4584] TS dbb72012a2f44c669051a4420d46daf6 (127.0.98.1:38385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b33889624f4442bb9d1258929b3422e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38385: connect: Connection refused (error 111)
I20260620 14:33:45.768203 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:45.768114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36291 (local address 127.0.0.1:46448)
0620 14:33:45.768150 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:45.768172 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:33:45.773124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13100
I20260620 14:33:45.778380 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:33:46.122972 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:33:46.123066 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:33:46.467911 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:33:46.468 UTC [12167] LOG: received smart shutdown request
2026-06-20 14:33:46.471 UTC [12167] LOG: background worker "logical replication launcher" (PID 12176) exited with exit code 1
2026-06-20 14:33:46.471 UTC [12171] LOG: shutting down
2026-06-20 14:33:46.635 UTC [12167] LOG: database system is shut down
2026-06-20T14:33:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47479 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:46 dist-test-slave-tc2g krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:46 dist-test-slave-tc2g krb5kdc[13669](info): set up 2 sockets
Jun 20 14:33:46 dist-test-slave-tc2g krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260620 14:33:47.793680 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:33:47 dist-test-slave-tc2g krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:33:47.912053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:47.992646 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:47.992852 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:47.992874 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:47.994078 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:47.994124 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:47.994138 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:47.994151 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:47.994207 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:47.995719 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:47.995966 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:47.996222 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:48.001020 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:48.001027 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:48.001027 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:48.001250 13682 server_base.cc:1034] running on GCE node
I20260620 14:33:48.001579 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:48.001880 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:48.003084 13682 hybrid_clock.cc:648] HybridClock initialized: now 1781966028003068 us; error 27 us; skew 500 ppm
Jun 20 14:33:47 dist-test-slave-tc2g krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:48.005651 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:48.006034 13682 webserver.cc:460] Webserver started at http://127.0.98.62:44609/ using document root <none> and password file <none>
I20260620 14:33:48.006230 13682 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:48.006273 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:48.006403 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:48.007442 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "f5976a2cb5a14124aa3187c43fb8a9df"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "acbb6601c064352bd86abae51a4d511f"
server_key_iv: "7dba4dbea93ae3150491f9ff8a5fff55"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.007740 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5976a2cb5a14124aa3187c43fb8a9df"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "acbb6601c064352bd86abae51a4d511f"
server_key_iv: "7dba4dbea93ae3150491f9ff8a5fff55"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.008975 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:33:48.009776 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:48.009948 13682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:48.010035 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "f5976a2cb5a14124aa3187c43fb8a9df"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "acbb6601c064352bd86abae51a4d511f"
server_key_iv: "7dba4dbea93ae3150491f9ff8a5fff55"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.010095 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:48.021606 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:48.023612 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:48.023727 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:48.027604 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33407
I20260620 14:33:48.027668 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33407 every 8 connection(s)
I20260620 14:33:48.028036 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:48.028934 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:48.030674 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:48.036936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13682
I20260620 14:33:48.037062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:33:48.037271 392 external_mini_cluster.cc:1428] Setting key 86914c2bea4e1f01f24090cf30677b35
I20260620 14:33:48.040529 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.031375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.0.1:53982)
0620 14:33:48.031642 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.031669 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.032172 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.032432 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.032435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.032676 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.032830 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.032833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.033840 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.033841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.034534 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.034536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.034984 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.034984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.035040 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.035566 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.035575 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.036921 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.039091 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.039097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.039099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.039287 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.039540 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.039541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.039541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.039630 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.040060 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.040065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.040124 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.040205 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.040329 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260620 14:33:48.040612 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.031383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.62:33407)
0620 14:33:48.031572 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.031581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.031764 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:33:48.032305 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.032307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.032324 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.032406 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.032918 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.033763 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.034608 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.034910 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.034993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.037005 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.037013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.037013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.037023 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.038993 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.039383 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.039386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.039386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.039472 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.039696 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.039698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.039698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.039986 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.040072 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.040257 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.040416 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260620 14:33:48.041527 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.041883 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.041770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35758)
0620 14:33:48.041818 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.041865 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:33:48.042063 13701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
I20260620 14:33:48.042464 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.042400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35772)
0620 14:33:48.042426 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.042448 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 14:33:48.042584 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:33:48.060271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36193
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:48.091816 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } attempt: 1
I20260620 14:33:48.092322 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.092186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35784)
0620 14:33:48.092251 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.092293 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:33:48.092646 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.092553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35796)
0620 14:33:48.092597 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.092623 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:33:48.092736 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
W20260620 14:33:48.144097 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:48.144330 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:48.144354 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:48.145682 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:48.145732 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:48.145748 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:48.145812 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:48.145839 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:48.147572 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:48.147861 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:48.148118 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:48.153602 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:48.153673 13757 server_base.cc:1034] running on GCE node
W20260620 14:33:48.153680 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:48.153684 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:48.154117 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:48.154449 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:48.155598 13757 hybrid_clock.cc:648] HybridClock initialized: now 1781966028155582 us; error 35 us; skew 500 ppm
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:48.158201 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:33:48.158707 13757 webserver.cc:460] Webserver started at http://127.0.98.61:43603/ using document root <none> and password file <none>
I20260620 14:33:48.158969 13757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:48.159022 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:48.159192 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:48.160300 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data/instance:
uuid: "df5132d98f894d1eaab80b4afc0c018c"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "665c5c5fc7ac095b19650355f527f4ed"
server_key_iv: "d5801d3fa398398bb1490e1d0f578c31"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.160666 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal/instance:
uuid: "df5132d98f894d1eaab80b4afc0c018c"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "665c5c5fc7ac095b19650355f527f4ed"
server_key_iv: "d5801d3fa398398bb1490e1d0f578c31"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.160864 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } attempt: 2
I20260620 14:33:48.161242 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.161122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35800)
0620 14:33:48.161178 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.161214 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:33:48.161588 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.161491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35814)
0620 14:33:48.161536 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.161561 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:33:48.161672 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36193: connect: Connection refused (error 111)
I20260620 14:33:48.162345 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:33:48.163164 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:48.163347 13757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:48.163445 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "df5132d98f894d1eaab80b4afc0c018c"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "665c5c5fc7ac095b19650355f527f4ed"
server_key_iv: "d5801d3fa398398bb1490e1d0f578c31"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.163532 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:48.176375 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:48.178294 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:48.178457 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:48.182548 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36193
I20260620 14:33:48.182610 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36193 every 8 connection(s)
I20260620 14:33:48.182979 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:33:48.183823 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:48.184839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13757
I20260620 14:33:48.184942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-1/wal/instance
I20260620 14:33:48.185115 392 external_mini_cluster.cc:1428] Setting key 4c767675ed862371334f297fdf0ddec7
I20260620 14:33:48.185887 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:33:48.196059 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.186755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.62:33407)
0620 14:33:48.186828 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.186831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.187066 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:33:48.187499 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.187501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.187518 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.187605 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.188480 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.189016 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.190679 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.190878 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.190933 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.193353 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.193355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.193355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.193364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.194714 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.195080 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.195081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.195081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.195153 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.195396 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.195396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.195396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.195496 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.195546 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.195821 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.195910 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:33:48.196151 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.186681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.0.1:53992)
0620 14:33:48.186929 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.186971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.187413 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.187672 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.187675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.188088 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.188385 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.188392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.189125 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.189126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.190545 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.190548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.190966 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.190967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.191119 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.191688 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.191711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.193254 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.194793 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.194799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.194801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.195015 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.195236 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.195237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.195237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.195337 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.195551 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.195555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.195665 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.195784 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.195939 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":147,"threads_started":1}
I20260620 14:33:48.196739 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:33:48.204332 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.197062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.61:36193)
0620 14:33:48.197378 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.197382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.197397 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:33:48.197413 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.197414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.197466 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.197539 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.197807 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.198483 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.199232 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.199525 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.199613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.201608 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.201618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.201618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.201628 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.203110 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.203465 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.203469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.203469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.203523 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.203758 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.203759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.203759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.203900 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.203997 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.204114 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.204233 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260620 14:33:48.204377 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.197024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35820)
0620 14:33:48.197075 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.197087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.197127 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.197548 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.197549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.197679 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.197720 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.197721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.198578 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.198579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.199137 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.199139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.199611 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.199612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.199675 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.200134 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.200138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.201517 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.203210 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.203210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.203211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.203374 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.203590 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.203591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.203591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.203644 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.203971 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.203972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.204014 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.204084 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.204201 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:33:48.205036 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.205410 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.205270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54442)
0620 14:33:48.205320 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.205387 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:33:48.205596 13774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
I20260620 14:33:48.206002 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.205897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54458)
0620 14:33:48.205962 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.205985 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:33:48.206146 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:33:48.212189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46175
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:48.252839 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } attempt: 3
I20260620 14:33:48.255402 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } attempt: 1
I20260620 14:33:48.255826 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.255702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54464)
0620 14:33:48.255758 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.255795 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:33:48.256296 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.256198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54470)
0620 14:33:48.256242 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.256275 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:33:48.256466 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:33:48.259999 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.253245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35828)
0620 14:33:48.253296 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.253315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.253380 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.253560 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.253561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.253729 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.253796 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.253798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.254501 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.254502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.255013 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.255014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.255451 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.255453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.255567 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.256098 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.256104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.257472 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.258932 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.258933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.258934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.259112 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.259326 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.259327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.259327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.259387 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.259665 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.259666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.259713 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.259798 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.259904 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:33:48.259997 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.253319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.0.98.61:36193)
0620 14:33:48.253392 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.253394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.253405 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:33:48.253460 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.253461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.253475 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.253555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.253882 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.254400 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.255109 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.255325 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.255415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.257544 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.257545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.257546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.257553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.258843 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.259198 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.259200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.259200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.259258 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.259477 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.259478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.259479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.259591 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.259662 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.259822 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.259904 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:33:48.260464 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.260798 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.260695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54474)
0620 14:33:48.260745 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.260776 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:33:48.261102 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.261046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54476)
0620 14:33:48.261068 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.261088 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 14:33:48.261202 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
I20260620 14:33:48.292361 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } attempt: 1
I20260620 14:33:48.292894 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.292760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54478)
0620 14:33:48.292827 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.292867 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:33:48.293277 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.293195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54490)
0620 14:33:48.293236 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.293261 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:33:48.293385 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
W20260620 14:33:48.296267 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:48.296453 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:48.296476 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:48.297699 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:48.297740 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:48.297781 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:48.297811 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:48.297828 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:48.299603 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33407,127.0.98.61:36193,127.0.98.60:46175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:48.299854 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:48.300084 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:48.304714 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:48.304687 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:48.304692 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:48.304971 13832 server_base.cc:1034] running on GCE node
I20260620 14:33:48.305203 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:48.305572 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:48.306725 13832 hybrid_clock.cc:648] HybridClock initialized: now 1781966028306710 us; error 30 us; skew 500 ppm
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:48.309149 13832 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:33:48.309669 13832 webserver.cc:460] Webserver started at http://127.0.98.60:43009/ using document root <none> and password file <none>
I20260620 14:33:48.309916 13832 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:48.309962 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:48.310127 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:48.311178 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data/instance:
uuid: "aaaa4e82f8df4f7ba81131104d3ed671"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "df88471aaec8c965e39bf26cd92cc9cd"
server_key_iv: "c8e88881c6b8980ce24b67cd22062c87"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.311542 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal/instance:
uuid: "aaaa4e82f8df4f7ba81131104d3ed671"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "df88471aaec8c965e39bf26cd92cc9cd"
server_key_iv: "c8e88881c6b8980ce24b67cd22062c87"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.312888 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:33:48.313627 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:48.313817 13832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:48.313903 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "aaaa4e82f8df4f7ba81131104d3ed671"
format_stamp: "Formatted at 2026-06-20 14:33:48 on dist-test-slave-tc2g"
server_key: "df88471aaec8c965e39bf26cd92cc9cd"
server_key_iv: "c8e88881c6b8980ce24b67cd22062c87"
server_key_version: "encryptionkey@0"
I20260620 14:33:48.314002 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:48.324611 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } attempt: 2
I20260620 14:33:48.324997 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.324876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54500)
0620 14:33:48.324935 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.324971 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:33:48.325403 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.325305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54512)
0620 14:33:48.325350 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.325375 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:33:48.325573 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46175: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46175: connect: Connection refused (error 111)
I20260620 14:33:48.325629 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:48.327483 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:48.327636 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:48.331409 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46175
I20260620 14:33:48.331471 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46175 every 8 connection(s)
I20260620 14:33:48.331925 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:33:48.332865 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:48.334453 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.337055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13832
I20260620 14:33:48.337153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965876904925-392-0/minicluster-data/master-2/wal/instance
I20260620 14:33:48.337339 392 external_mini_cluster.cc:1428] Setting key f5a26d3084e2e34fc9b1d846f306e3e7
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:33:48.344141 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.335137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.0.98.62:33407)
0620 14:33:48.335217 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.335219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.335371 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:33:48.335801 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.335802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.335811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.335863 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.336419 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.336908 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.338426 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.338618 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.338672 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.341168 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.341169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.341169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.341176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.342669 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.343160 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.343161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.343162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.343213 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.343507 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.343507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.343508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.343608 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.343688 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.343968 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.344052 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:33:48.344324 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.335098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.0.1:53994)
0620 14:33:48.335256 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.335291 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.335721 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.335917 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.335920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.336157 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.336347 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.336351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.337000 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.337001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.338335 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.338339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.338704 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.338706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.338862 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.339379 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.339387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.341074 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.342752 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.342759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.342761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.343057 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.343285 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.343286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.343286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.343425 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.343680 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.343687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.343818 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.343935 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.344083 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 14:33:48.344893 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.346818 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.337777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.0.98.60:46175)
0620 14:33:48.337965 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.337969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.337984 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:33:48.338003 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.338004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.338021 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.338078 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.338777 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.339437 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.340345 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.340822 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.340907 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.343576 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.343582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.343582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.343589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.345641 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.346038 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.346042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.346042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.346100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.346297 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.346298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.346298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.346440 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.346606 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.346648 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.346720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:48.352166 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.345202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.0.98.61:36193)
0620 14:33:48.345242 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.345244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.345274 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:33:48.345295 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.345295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.345303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.345342 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.345745 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.346391 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.347232 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.347504 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.347579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.349302 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.349303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.349304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.349311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.351061 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.351392 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.351394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.351394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.351461 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.351673 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.351674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.351674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.351782 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.351832 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.352017 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.352076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:33:48.352201 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.345139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36193 (local address 127.0.0.1:35844)
0620 14:33:48.345178 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.345188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.345221 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.345370 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.345371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.345634 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.345672 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.345674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.346462 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.346463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.347139 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.347141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.347587 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.347589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.347648 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.348046 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.348051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.349226 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.351151 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.351152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.351152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.351315 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.351535 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.351536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.351537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.351604 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.351856 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.351857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.351920 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.352001 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.352084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:33:48.352731 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:48.355198 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.347622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.0.98.62:33407)
0620 14:33:48.347660 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.347661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.347670 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:48.347722 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.347722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.347732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.347786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.348245 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.348901 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.349878 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.350077 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.350132 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.352729 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.352730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.352730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.352737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.354159 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.354511 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.354511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.354511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.354557 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.354736 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.354737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.354737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.354824 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.354955 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.355037 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.355111 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:33:48.359565 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.353007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54526)
0620 14:33:48.353218 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.353226 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.353289 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.353401 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.353401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.353553 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.353578 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.353580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.354187 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.354187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.354948 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.354950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.355417 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.355418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.355476 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.355804 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.355809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.356990 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.358582 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.358582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.358583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.358741 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.358967 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.358968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.358968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.359021 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.359261 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.359262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.359298 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.359365 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.359457 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260620 14:33:48.359616 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.353054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.0.98.60:46175)
0620 14:33:48.353094 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.353096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.353276 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:33:48.353341 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.353342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.353351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.353398 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.353659 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.354102 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.355029 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.355342 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.355414 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.357073 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.357075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.357075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.357081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.358482 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.358821 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.358822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.358822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.358888 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.359087 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.359088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.359088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.359192 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.359269 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.359392 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.359503 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":20}
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:33:48.361227 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: Bootstrap starting.
I20260620 14:33:48.362401 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:48.362705 13900 log.cc:824] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:48.363533 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.356200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.0.98.61:36193)
0620 14:33:48.356238 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.356239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.356419 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:33:48.356489 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.356490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.356500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.356554 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.357068 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.357577 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.358215 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.358416 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.358489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.360794 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.360795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.360796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.360802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.362446 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.362795 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.362796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.362796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.362852 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.363038 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.363039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.363039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.363178 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.363341 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.363414 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.363472 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:33:48.363619 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: No bootstrap required, opened a new log
I20260620 14:33:48.365388 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:33:48.365890 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.366036 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:48.366075 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaa4e82f8df4f7ba81131104d3ed671, State: Initialized, Role: FOLLOWER
I20260620 14:33:48.366206 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.366555 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:48.366516 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.366614 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:48.369177 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } attempt: 2
W20260620 14:33:48.369225 13924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:48.370436 13924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:48.370110 13832 master_runner.cc:186] Error getting master registration for 127.0.98.62:33407: OK, instance_id { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" instance_seqno: 1781966028026509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:33:48.372694 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:33:48.376434 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.366863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.0.98.62:33407)
0620 14:33:48.366919 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.366922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.367390 (+ 468us) server_negotiation.cc:407] Connection header received
0620 14:33:48.367495 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.367496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.367507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.367537 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.368353 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.369090 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.370715 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.371002 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.371069 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.373460 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.373461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.373462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.373468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.375169 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.375562 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.375563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.375563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.375628 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.375826 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.375827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.375827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.375948 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.376027 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.376220 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.376304 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:33:48.376674 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.367200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.0.98.61:36193)
0620 14:33:48.367243 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.367245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.367387 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:33:48.367492 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.367493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.367503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.367535 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.368237 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.368968 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.370085 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.370348 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.370413 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.372325 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.372334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.372335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.372342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.374989 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.375437 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.375438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.375439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.375495 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.375704 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.375705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.375705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.375837 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.375949 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.376492 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.376584 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:33:48.378432 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.366824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.0.98.60:46175)
0620 14:33:48.367034 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.367036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.367146 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:33:48.367315 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.367316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.367329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.367403 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.368108 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.370291 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.371479 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.371782 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.372123 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.373341 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.373342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.373343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.373350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.375723 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.377521 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.377522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.377523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.377579 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.377786 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.377787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.377787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.377933 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.378030 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.378174 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.378255 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260620 14:33:48.381191 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.369410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54550)
0620 14:33:48.369536 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.369545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.369582 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.372956 (+ 3374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.372957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.373136 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.373181 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.373183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.374200 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.374201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.374976 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.374979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.376205 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.376206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.376267 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.376996 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.377001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.378574 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.380322 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.380322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.380323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.380484 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.380682 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.380682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.380683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.380729 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.380981 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.380982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.381016 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.381065 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.381130 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 14:33:48.381279 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.369663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.0.98.60:46175)
0620 14:33:48.372787 (+ 3124us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.372790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.372797 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:33:48.372818 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.372819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.372840 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.373251 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.373279 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.374084 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.375805 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.376110 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.376413 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.378670 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.378671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.378671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.378678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.380231 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.380553 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.380554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.380554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.380613 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.380784 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.380785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.380785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.380918 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.380987 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.381098 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.381171 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3103,"thread_start_us":62,"threads_started":1}
I20260620 14:33:48.382859 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: Bootstrap starting.
I20260620 14:33:48.383617 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:48.383930 13750 log.cc:824] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:48.384707 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: No bootstrap required, opened a new log
I20260620 14:33:48.386756 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.386909 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:48.386945 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5976a2cb5a14124aa3187c43fb8a9df, State: Initialized, Role: FOLLOWER
I20260620 14:33:48.387063 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.387368 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:48.387336 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.387441 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:33:48.389173 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:48.389274 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:48.389914 13682 master_runner.cc:186] Error getting master registration for 127.0.98.61:36193: OK, instance_id { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" instance_seqno: 1781966028181248 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:33:48.392143 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:33:48.416725 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } attempt: 3
Jun 20 14:33:48 dist-test-slave-tc2g krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:33:48.424154 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.417097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46175 (local address 127.0.0.1:54560)
0620 14:33:48.417157 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:48.417170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:48.417212 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:48.417397 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:48.417398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:48.417595 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:48.417637 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.417639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.418393 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.418394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.418886 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:48.418888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.419296 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.419297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:48.419363 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.419851 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:48.419857 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:48.421378 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:48.423100 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.423101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.423102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.423275 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.423493 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:48.423494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:48.423495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:48.423558 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:48.423809 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:48.423810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:48.423856 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:33:48.423948 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:33:48.424051 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:33:48.424154 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:48.417137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.0.98.60:46175)
0620 14:33:48.417245 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:48.417247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:48.417257 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:48.417285 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:48.417291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:48.417299 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:48.417366 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:48.417732 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.418294 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.418982 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:48.419207 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:48.419296 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:48.421529 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:48.421531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:48.421531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:48.421539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:48.422997 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.423360 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.423361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.423362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.423426 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:48.423622 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:48.423622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:48.423623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:48.423741 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:48.423817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:48.423972 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:33:48.424053 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:33:48.425869 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: Bootstrap starting.
I20260620 14:33:48.426532 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:48.426831 13825 log.cc:824] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:48.427604 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: No bootstrap required, opened a new log
I20260620 14:33:48.429106 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.429241 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:48.429281 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5132d98f894d1eaab80b4afc0c018c, State: Initialized, Role: FOLLOWER
I20260620 14:33:48.429486 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.429764 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.429843 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:48.429896 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:33:48.432286 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:48.432392 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:48.433799 13757 master_runner.cc:263] No leader master found from master df5132d98f894d1eaab80b4afc0c018c
I20260620 14:33:48.433844 13757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:33:48.479076 13907 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:48.479195 13907 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.479442 13907 leader_election.cc:288] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5976a2cb5a14124aa3187c43fb8a9df (127.0.98.62:33407), df5132d98f894d1eaab80b4afc0c018c (127.0.98.61:36193)
I20260620 14:33:48.479913 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5132d98f894d1eaab80b4afc0c018c" is_pre_election: true
I20260620 14:33:48.480065 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaa4e82f8df4f7ba81131104d3ed671 in term 0.
I20260620 14:33:48.480366 13851 leader_election.cc:302] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaa4e82f8df4f7ba81131104d3ed671, df5132d98f894d1eaab80b4afc0c018c; no voters:
I20260620 14:33:48.481087 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" is_pre_election: true
I20260620 14:33:48.481211 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaa4e82f8df4f7ba81131104d3ed671 in term 0.
I20260620 14:33:48.481570 13907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:48.481634 13907 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:48.481657 13907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:48.482549 13907 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.482659 13907 leader_election.cc:288] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [CANDIDATE]: Term 1 election: Requested vote from peers f5976a2cb5a14124aa3187c43fb8a9df (127.0.98.62:33407), df5132d98f894d1eaab80b4afc0c018c (127.0.98.61:36193)
I20260620 14:33:48.482936 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5132d98f894d1eaab80b4afc0c018c"
I20260620 14:33:48.483006 13799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:48.484215 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaa4e82f8df4f7ba81131104d3ed671 in term 1.
I20260620 14:33:48.484436 13851 leader_election.cc:302] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaa4e82f8df4f7ba81131104d3ed671, df5132d98f894d1eaab80b4afc0c018c; no voters:
I20260620 14:33:48.484695 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5976a2cb5a14124aa3187c43fb8a9df"
I20260620 14:33:48.484767 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:48.485664 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaa4e82f8df4f7ba81131104d3ed671 in term 1.
I20260620 14:33:48.485882 13907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:48.486044 13907 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [term 1 LEADER]: Becoming Leader. State: Replica: aaaa4e82f8df4f7ba81131104d3ed671, State: Running, Role: LEADER
I20260620 14:33:48.486125 13907 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } }
I20260620 14:33:48.486472 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaaa4e82f8df4f7ba81131104d3ed671. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.486536 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:48.486919 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:48.487208 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:48.489321 13799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [term 1 FOLLOWER]: Refusing update from remote peer aaaa4e82f8df4f7ba81131104d3ed671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:48.489681 13907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:48.490803 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaaa4e82f8df4f7ba81131104d3ed671. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.490880 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:48.491282 13724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [term 1 FOLLOWER]: Refusing update from remote peer aaaa4e82f8df4f7ba81131104d3ed671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:48.491582 13907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:48.492705 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaaa4e82f8df4f7ba81131104d3ed671. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.492801 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:48.494024 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.494105 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:48.494621 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.494694 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:48.494921 13958 mvcc.cc:205] Tried to move back new op lower bound from 7298932852690296832 to 7298932852679671808. Current Snapshot: MvccSnapshot[applied={T|T < 7298932852690296832}]
I20260620 14:33:48.495128 13961 mvcc.cc:205] Tried to move back new op lower bound from 7298932852690296832 to 7298932852679671808. Current Snapshot: MvccSnapshot[applied={T|T < 7298932852690296832}]
I20260620 14:33:48.495867 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.495944 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:48.496282 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5976a2cb5a14124aa3187c43fb8a9df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } peers { permanent_uuid: "df5132d98f894d1eaab80b4afc0c018c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36193 } } peers { permanent_uuid: "aaaa4e82f8df4f7ba81131104d3ed671" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46175 } } }
I20260620 14:33:48.496359 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:48.496973 13959 mvcc.cc:205] Tried to move back new op lower bound from 7298932852690296832 to 7298932852679671808. Current Snapshot: MvccSnapshot[applied={T|T < 7298932852690296832}]
I20260620 14:33:48.497895 13957 catalog_manager.cc:1347] Generated new cluster ID: cb7956068a754c38ad263183bcc42f5b
I20260620 14:33:48.497944 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:48.514763 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:48.515754 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:48.523751 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaaa4e82f8df4f7ba81131104d3ed671: Generated new TSK 0
I20260620 14:33:48.525030 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:33:49.389731 13939 catalog_manager.cc:1259] Loaded cluster ID: cb7956068a754c38ad263183bcc42f5b
I20260620 14:33:49.389806 13939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: loading cluster ID for follower catalog manager: success
I20260620 14:33:49.390640 13939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: acquiring CA information for follower catalog manager: success
I20260620 14:33:49.390969 13939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5976a2cb5a14124aa3187c43fb8a9df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:33:49.432857 13953 catalog_manager.cc:1259] Loaded cluster ID: cb7956068a754c38ad263183bcc42f5b
I20260620 14:33:49.432943 13953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: loading cluster ID for follower catalog manager: success
I20260620 14:33:49.433794 13953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: acquiring CA information for follower catalog manager: success
I20260620 14:33:49.434126 13953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df5132d98f894d1eaab80b4afc0c018c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:33:49.591285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13682
I20260620 14:33:49.595960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13757
I20260620 14:33:49.600695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13832
2026-06-20T14:33:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2854 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:49 dist-test-slave-tc2g krb5kdc[13973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:49 dist-test-slave-tc2g krb5kdc[13973](info): set up 2 sockets
Jun 20 14:33:49 dist-test-slave-tc2g krb5kdc[13973](info): commencing operation
krb5kdc: starting...
W20260620 14:33:50.649580 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:33:50 dist-test-slave-tc2g krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:33:50.764344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:50.844035 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:50.844280 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:50.844317 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:50.845541 13986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:50.845603 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:50.845628 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:50.845654 13986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:50.845685 13986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:50.847163 13986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:50.847447 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:50.847674 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:50.852556 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:50.852653 13986 server_base.cc:1034] running on GCE node
W20260620 14:33:50.852543 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:50.852536 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:50.853399 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:50.853729 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:50.854892 13986 hybrid_clock.cc:648] HybridClock initialized: now 1781966030854885 us; error 33 us; skew 500 ppm
Jun 20 14:33:50 dist-test-slave-tc2g krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:50.862119 13986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:50.862556 13986 webserver.cc:460] Webserver started at http://127.0.98.62:34221/ using document root <none> and password file <none>
I20260620 14:33:50.862790 13986 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:50.862860 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:50.863011 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:50.864075 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "8bb46cb48ca14051a0a3ac08051a0f6b"
format_stamp: "Formatted at 2026-06-20 14:33:50 on dist-test-slave-tc2g"
server_key: "87daf4bc77b031a58786467f4c83ba1f"
server_key_iv: "0205f45219746c3b0cc3f2133448d813"
server_key_version: "encryptionkey@0"
I20260620 14:33:50.864435 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "8bb46cb48ca14051a0a3ac08051a0f6b"
format_stamp: "Formatted at 2026-06-20 14:33:50 on dist-test-slave-tc2g"
server_key: "87daf4bc77b031a58786467f4c83ba1f"
server_key_iv: "0205f45219746c3b0cc3f2133448d813"
server_key_version: "encryptionkey@0"
I20260620 14:33:50.865794 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:33:50.866524 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:50.866693 13986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:33:50.866788 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "8bb46cb48ca14051a0a3ac08051a0f6b"
format_stamp: "Formatted at 2026-06-20 14:33:50 on dist-test-slave-tc2g"
server_key: "87daf4bc77b031a58786467f4c83ba1f"
server_key_iv: "0205f45219746c3b0cc3f2133448d813"
server_key_version: "encryptionkey@0"
I20260620 14:33:50.866870 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:50.883411 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:50.885214 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:50.885378 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:50.889292 13986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45867
I20260620 14:33:50.889341 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45867 every 8 connection(s)
I20260620 14:33:50.889722 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:50.890578 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:50.892833 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b: Bootstrap starting.
I20260620 14:33:50.893442 14054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:50.893744 14054 log.cc:824] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:50.894410 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b: No bootstrap required, opened a new log
I20260620 14:33:50.895974 14054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } }
I20260620 14:33:50.896109 14054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:50.896132 14054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb46cb48ca14051a0a3ac08051a0f6b, State: Initialized, Role: FOLLOWER
I20260620 14:33:50.896302 14054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } }
I20260620 14:33:50.896382 14054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:50.896436 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:50.896497 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:50.897009 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } }
I20260620 14:33:50.897122 14054 leader_election.cc:302] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bb46cb48ca14051a0a3ac08051a0f6b; no voters:
I20260620 14:33:50.897308 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:50.897346 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:50.897524 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [term 1 LEADER]: Becoming Leader. State: Replica: 8bb46cb48ca14051a0a3ac08051a0f6b, State: Running, Role: LEADER
I20260620 14:33:50.897611 14054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:50.897650 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } }
I20260620 14:33:50.898234 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bb46cb48ca14051a0a3ac08051a0f6b. Latest consensus state: current_term: 1 leader_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } } }
I20260620 14:33:50.898334 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:50.898466 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb46cb48ca14051a0a3ac08051a0f6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45867 } } }
I20260620 14:33:50.898594 14067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:50.898658 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:50.898741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 13986
I20260620 14:33:50.898828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:33:50.899016 392 external_mini_cluster.cc:1428] Setting key adf0de965d9a1b8fadac6c5566a99035
I20260620 14:33:50.899549 14067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:50.901823 14067 catalog_manager.cc:1347] Generated new cluster ID: cbc869a1d370429fa60b78f657c61f12
I20260620 14:33:50.901872 14067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:33:50 dist-test-slave-tc2g krb5kdc[13973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:50.909899 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.899476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.0.98.62:45867)
0620 14:33:50.899633 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:50.899643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:50.900183 (+ 540us) server_negotiation.cc:407] Connection header received
0620 14:33:50.900327 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:50.900335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:50.900392 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:50.900548 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:50.901398 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.902835 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.903652 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.904084 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.904216 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.906653 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:50.906657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:50.906658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:50.906669 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:50.908318 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:50.908711 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:50.908714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:50.908714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:50.908810 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:50.909024 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:50.909025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:50.909025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:50.909249 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:50.909448 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:50.909557 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:33:50.909698 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260620 14:33:50.910897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:33:50.924507 14067 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:50.925135 14067 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:50.928656 14067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bb46cb48ca14051a0a3ac08051a0f6b: Generated new TSK 0
I20260620 14:33:50.928848 14067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:33:50.964797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 13986
2026-06-20T14:33:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1362 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:51 dist-test-slave-tc2g krb5kdc[14081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:51 dist-test-slave-tc2g krb5kdc[14081](info): set up 2 sockets
Jun 20 14:33:51 dist-test-slave-tc2g krb5kdc[14081](info): commencing operation
krb5kdc: starting...
W20260620 14:33:51.993592 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:33:52.108408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:52.188472 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:52.188735 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:52.188773 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:52.189989 14094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:52.190063 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:52.190088 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:52.190109 14094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:52.190138 14094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:52.191627 14094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:52.191927 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:52.192164 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:52.196938 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.196903 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:52.197007 14094 server_base.cc:1034] running on GCE node
W20260620 14:33:52.196903 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:52.197408 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:52.197763 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:52.198918 14094 hybrid_clock.cc:648] HybridClock initialized: now 1781966032198906 us; error 26 us; skew 500 ppm
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:52.201251 14094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:52.201828 14094 webserver.cc:460] Webserver started at http://127.0.98.62:34415/ using document root <none> and password file <none>
I20260620 14:33:52.202072 14094 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:52.202143 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:52.202288 14094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:52.203408 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "80ef4d03b7d44720827181abb3c3e24f"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "6b393122ae0c85a4b59fbd5e457bcf44"
server_key_iv: "576066cdd0c0a074b3d3076570d0df5d"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.203779 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "80ef4d03b7d44720827181abb3c3e24f"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "6b393122ae0c85a4b59fbd5e457bcf44"
server_key_iv: "576066cdd0c0a074b3d3076570d0df5d"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.205099 14094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:33:52.205942 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.206115 14094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.206214 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "80ef4d03b7d44720827181abb3c3e24f"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "6b393122ae0c85a4b59fbd5e457bcf44"
server_key_iv: "576066cdd0c0a074b3d3076570d0df5d"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.206301 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:52.228753 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:52.230762 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:52.230933 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:52.235013 14094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34007
I20260620 14:33:52.235046 14161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34007 every 8 connection(s)
I20260620 14:33:52.235433 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:52.236428 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:52.238778 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap starting.
I20260620 14:33:52.239367 14162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:52.239650 14162 log.cc:824] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:52.240401 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: No bootstrap required, opened a new log
I20260620 14:33:52.242219 14162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:52.242357 14162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:52.242379 14162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Initialized, Role: FOLLOWER
I20260620 14:33:52.242535 14162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:52.242600 14162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:52.242623 14162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:52.242722 14162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:52.243218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14094
I20260620 14:33:52.243351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:33:52.243582 392 external_mini_cluster.cc:1428] Setting key 41131b088426af8e9fb597746f51e56e
I20260620 14:33:52.243775 14162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:52.243904 14162 leader_election.cc:302] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ef4d03b7d44720827181abb3c3e24f; no voters:
I20260620 14:33:52.244107 14162 leader_election.cc:288] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:52.244160 14165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:52.244287 14165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 LEADER]: Becoming Leader. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Running, Role: LEADER
I20260620 14:33:52.244366 14165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:52.244422 14162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:52.244694 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ef4d03b7d44720827181abb3c3e24f. Latest consensus state: current_term: 1 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:52.244764 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:52.244726 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:52.244963 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
W20260620 14:33:52.246335 14182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:52.246403 14182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:52.246486 14171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:52.246651 14171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:52.248026 14171 catalog_manager.cc:1347] Generated new cluster ID: 779e7d0890724e2ea4cc710c944f71ac
I20260620 14:33:52.248080 14171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:52.256239 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.244143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.98.62:34007)
0620 14:33:52.244810 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.244820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.244854 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:33:52.244972 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.244979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.245036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.245174 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:52.246106 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.247530 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.248608 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.249262 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.249344 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.252294 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:52.252298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:52.252299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:52.252311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:52.254428 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.254828 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.254832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.254832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.254957 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.255167 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.255168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.255168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.255495 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:52.255668 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.255744 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.256012 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":110,"threads_started":1}
I20260620 14:33:52.307500 14171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:52.308039 14171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:52.318189 14171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Generated new TSK 0
I20260620 14:33:52.318380 14171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:33:52.358361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:52.441738 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:52.441996 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:52.442022 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:52.443138 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:52.443174 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:52.443199 14188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:33:52.444617 14188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:52.444792 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:52.444963 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:52.449975 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.450040 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.450033 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:52.450208 14188 server_base.cc:1034] running on GCE node
I20260620 14:33:52.450469 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:52.450729 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:52.451869 14188 hybrid_clock.cc:648] HybridClock initialized: now 1781966032451844 us; error 41 us; skew 500 ppm
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:52.454479 14188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:33:52.454919 14188 webserver.cc:460] Webserver started at http://127.0.98.1:37893/ using document root <none> and password file <none>
I20260620 14:33:52.455163 14188 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:52.455210 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:52.455364 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:52.456403 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "c245c8b1a20245cdad4618feedff7cee"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "16fd332c9759b9afc5cc8e0376e0d2e8"
server_key_iv: "f5f24807534e260e75954bdafe0b31bc"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.456794 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c245c8b1a20245cdad4618feedff7cee"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "16fd332c9759b9afc5cc8e0376e0d2e8"
server_key_iv: "f5f24807534e260e75954bdafe0b31bc"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.458225 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:33:52.458949 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.459336 14188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:52.459431 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "c245c8b1a20245cdad4618feedff7cee"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "16fd332c9759b9afc5cc8e0376e0d2e8"
server_key_iv: "f5f24807534e260e75954bdafe0b31bc"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.459487 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:52.470714 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:52.473071 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:52.473186 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:52.473455 14188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:52.473737 14188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:52.473768 14188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.473789 14188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:52.473802 14188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.479288 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40175
I20260620 14:33:52.479363 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40175 every 8 connection(s)
I20260620 14:33:52.479686 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:52.483582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14188
I20260620 14:33:52.483714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:33:52.483902 392 external_mini_cluster.cc:1428] Setting key 3cd71906bd739385efe6a4295ccaf8c2
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:52.489480 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.480404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60831 (local address 127.0.98.62:34007)
0620 14:33:52.480486 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.480489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.480657 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:33:52.481135 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.481136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.481151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.481206 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:52.482195 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.482690 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.483641 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.483967 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.484043 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.486528 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:52.486530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:52.486530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:52.486538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:52.488041 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.488461 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.488462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.488463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.488523 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.488786 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.488787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.488787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.488902 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:52.488979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.489254 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.489341 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:33:52.489677 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.480358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:60831)
0620 14:33:52.480540 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.480575 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.481044 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.481265 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.481269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.481908 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:52.482118 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.482123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.482770 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.482771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.483561 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.483563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.484063 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.484064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.484212 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.484763 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:52.484771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:52.486431 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:52.488124 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.488131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.488133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.488370 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.488592 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.488593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.488594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.488712 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.488970 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:52.488977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:52.489123 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.489232 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.489398 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260620 14:33:52.490316 14317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:52.490442 14317 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:52.490633 14317 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:52.491231 14126 ts_manager.cc:194] Registered new tserver with Master: c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:33:52.492221 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:33:52.508213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:52.588593 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:52.588841 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:52.588881 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:52.590052 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:52.590126 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:52.590170 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:33:52.591670 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:52.591965 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:52.592199 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:52.597221 14322 server_base.cc:1034] running on GCE node
W20260620 14:33:52.597138 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.597159 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.597162 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:52.597616 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:52.597867 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:52.599050 14322 hybrid_clock.cc:648] HybridClock initialized: now 1781966032599039 us; error 30 us; skew 500 ppm
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:52.601511 14322 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:33:52.601945 14322 webserver.cc:460] Webserver started at http://127.0.98.2:36493/ using document root <none> and password file <none>
I20260620 14:33:52.602193 14322 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:52.602259 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:52.602428 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:52.603467 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "21451b18edca432291e22c3dac2ad168"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "3c2e8c213e7f40cd3dc22c7dd9b4e74c"
server_key_iv: "c62112a63a134b390c4fe0f099890de7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.603837 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21451b18edca432291e22c3dac2ad168"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "3c2e8c213e7f40cd3dc22c7dd9b4e74c"
server_key_iv: "c62112a63a134b390c4fe0f099890de7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.605165 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:33:52.605963 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.606133 14322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.606240 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "21451b18edca432291e22c3dac2ad168"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "3c2e8c213e7f40cd3dc22c7dd9b4e74c"
server_key_iv: "c62112a63a134b390c4fe0f099890de7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.606329 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:52.626904 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:52.628820 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:52.628995 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:52.629241 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:52.629587 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:52.629644 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.629685 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:52.629714 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.635609 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40731
I20260620 14:33:52.635658 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40731 every 8 connection(s)
I20260620 14:33:52.635973 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:52.643124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14322
I20260620 14:33:52.643251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:33:52.643492 392 external_mini_cluster.cc:1428] Setting key 1604a60b14556ae717e80657f39ecd66
I20260620 14:33:52.645622 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.636658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38437 (local address 127.0.98.62:34007)
0620 14:33:52.636737 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.636739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.636971 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:33:52.637487 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.637488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.637500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.637562 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:52.638483 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.638970 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.640004 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.640186 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.640236 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.642516 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:52.642518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:52.642518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:52.642525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:52.644083 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.644558 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.644560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.644560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.644618 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.644905 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.644906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.644907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.645057 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:52.645123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.645451 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.645534 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:33:52.645809 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.636650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:38437)
0620 14:33:52.636826 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.636869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.637359 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.637636 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.637639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.638207 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:52.638401 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.638406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.639087 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.639088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.639915 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.639917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.640261 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.640262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.640369 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.640820 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:52.640828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:52.642429 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:52.644160 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.644167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.644169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.644463 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.644693 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.644694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.644694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.644829 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.645114 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:52.645120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:52.645266 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.645370 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.645573 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260620 14:33:52.646229 14451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:52.646334 14451 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:52.646550 14451 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:52.646952 14126 ts_manager.cc:194] Registered new tserver with Master: 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
I20260620 14:33:52.647476 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38437
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:33:52.667948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:52.750566 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:52.750770 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:52.750794 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:52.752275 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:52.752316 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:52.752343 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:33:52.753880 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:52.754070 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:52.754251 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:52.759696 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.759639 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:52.759639 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:52.759644 14456 server_base.cc:1034] running on GCE node
I20260620 14:33:52.759994 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:52.760185 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:52.761313 14456 hybrid_clock.cc:648] HybridClock initialized: now 1781966032761303 us; error 33 us; skew 500 ppm
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:52.763916 14456 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:33:52.764341 14456 webserver.cc:460] Webserver started at http://127.0.98.3:43003/ using document root <none> and password file <none>
I20260620 14:33:52.764567 14456 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:52.764612 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:52.764778 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:52.765895 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "f3d2319e31784891a6a991a62f87ac09"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "9d13bd21896f54c62e5c669766b06a89"
server_key_iv: "237dfa1168d8f2b6201cd3d4dc5ab6c7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.766245 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f3d2319e31784891a6a991a62f87ac09"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "9d13bd21896f54c62e5c669766b06a89"
server_key_iv: "237dfa1168d8f2b6201cd3d4dc5ab6c7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.767525 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:33:52.768217 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.768388 14456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:52.768507 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "f3d2319e31784891a6a991a62f87ac09"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "9d13bd21896f54c62e5c669766b06a89"
server_key_iv: "237dfa1168d8f2b6201cd3d4dc5ab6c7"
server_key_version: "encryptionkey@0"
I20260620 14:33:52.768561 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:52.784765 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:52.786757 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:52.786931 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:52.787171 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:52.787495 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:52.787550 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.787590 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:52.787614 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:52.793650 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43019
I20260620 14:33:52.793731 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43019 every 8 connection(s)
I20260620 14:33:52.794062 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:33:52 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:52.802703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14456
I20260620 14:33:52.802726 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.794794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59827 (local address 127.0.98.62:34007)
0620 14:33:52.794874 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.794876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.795103 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:33:52.795589 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.795590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.795600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.795651 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:52.796539 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.797027 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.798072 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.798258 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.798309 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.800281 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:52.800282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:52.800282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:52.800289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:52.801489 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.801834 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.801834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.801835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.801883 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.802093 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.802093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.802094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.802188 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:52.802288 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.802507 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.802629 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:33:52.802796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:33:52.802819 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.794777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:59827)
0620 14:33:52.794983 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.795020 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.795500 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.795704 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.795707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.796263 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:52.796465 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.796470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.797113 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.797114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.797998 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.798000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.798327 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.798328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.798445 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.798886 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:52.798894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:52.800198 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:52.801554 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.801560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.801561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.801770 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.801945 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:52.801946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:52.801946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:52.802039 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:52.802277 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:52.802281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:52.802385 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.802479 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.802635 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260620 14:33:52.802999 392 external_mini_cluster.cc:1428] Setting key b739970ba3457eec04764cbd4c9a40a3
I20260620 14:33:52.803463 14585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:52.803587 14585 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:52.803776 14585 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:52.804188 14125 ts_manager.cc:194] Registered new tserver with Master: f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019)
I20260620 14:33:52.804581 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59827
I20260620 14:33:52.814692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:52.822016 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.816042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.98.62:34007)
0620 14:33:52.816117 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.816119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.816229 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:33:52.816267 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.816268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.816277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.816322 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:52.816852 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.817341 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.818134 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.818428 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.818483 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.819581 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:52.819582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:52.819582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:52.819588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:52.820960 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.821253 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.821253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.821254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.821301 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:52.821561 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:52.821562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:52.821562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:52.821701 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:52.821781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.821868 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.821927 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:33:52.823707 14125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:52.825467 14125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:52.832592 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.827937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40731 (local address 127.0.0.1:44800)
0620 14:33:52.828286 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.828333 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.828401 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.829036 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.829036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.829038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:52.829083 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.829084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.830017 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.830018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.831272 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.831274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.832275 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.832276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.832359 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.832374 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.832448 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.832512 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":67,"threads_started":1}
I20260620 14:33:52.832612 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.828072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.0.98.2:40731)
0620 14:33:52.828865 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.828870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.828879 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:52.828920 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.828922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.828935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.829039 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:52.829153 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.829914 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.831363 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.832186 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.832277 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.832359 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.832462 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.832505 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":103,"threads_started":1}
I20260620 14:33:52.832921 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.828141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.0.98.1:40175)
0620 14:33:52.828570 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.828576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.828586 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:52.828623 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.828625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.828640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.828722 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:52.828914 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.830222 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.831472 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.832296 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.832363 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.832414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.832802 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.832859 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":88,"threads_started":1}
I20260620 14:33:52.832736 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.827937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.0.1:33914)
0620 14:33:52.828285 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.828330 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.828399 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.828730 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.828733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.828735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:52.828808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.828823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.830319 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.830320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.831370 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.831372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.832453 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.832454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.832537 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.832541 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.832597 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.832650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":5,"thread_start_us":70,"threads_started":1}
I20260620 14:33:52.833611 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.828155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43019 (local address 127.0.0.1:59414)
0620 14:33:52.828317 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.828336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.828399 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.828777 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.828777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.828779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:52.828816 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.828818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.831463 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.831464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.832120 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.832121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.833359 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.833360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.833420 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.833425 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.833498 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.833536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260620 14:33:52.833856 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.828276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.98.3:43019)
0620 14:33:52.828505 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.828510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.828520 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:52.828571 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.828573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.828611 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.828693 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:52.828910 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.830203 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.832208 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.833202 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.833263 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.833324 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.833720 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.833767 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260620 14:33:52.833822 14251 tablet_service.cc:1467] Processing CreateTablet for tablet a5be536edddf4904b653be612f74a3f3 (DEFAULT_TABLE table=test-table [id=1cada87998ff47c893e502df8ff58889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:52.833843 14385 tablet_service.cc:1467] Processing CreateTablet for tablet a5be536edddf4904b653be612f74a3f3 (DEFAULT_TABLE table=test-table [id=1cada87998ff47c893e502df8ff58889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:52.834204 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5be536edddf4904b653be612f74a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:52.834232 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5be536edddf4904b653be612f74a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:52.834522 14519 tablet_service.cc:1467] Processing CreateTablet for tablet a5be536edddf4904b653be612f74a3f3 (DEFAULT_TABLE table=test-table [id=1cada87998ff47c893e502df8ff58889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:52.834824 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5be536edddf4904b653be612f74a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:52.836982 14602 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Bootstrap starting.
I20260620 14:33:52.837750 14602 tablet_bootstrap.cc:654] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:52.837805 14603 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Bootstrap starting.
I20260620 14:33:52.838086 14604 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Bootstrap starting.
I20260620 14:33:52.838431 14603 tablet_bootstrap.cc:654] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:52.838409 14602 log.cc:824] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:52.838871 14604 tablet_bootstrap.cc:654] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:52.839061 14603 log.cc:824] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:52.839460 14602 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: No bootstrap required, opened a new log
I20260620 14:33:52.839529 14604 log.cc:824] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:52.839547 14602 ts_tablet_manager.cc:1397] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:33:52.840027 14603 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: No bootstrap required, opened a new log
I20260620 14:33:52.840116 14603 ts_tablet_manager.cc:1397] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:52.840333 14604 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: No bootstrap required, opened a new log
I20260620 14:33:52.840413 14604 ts_tablet_manager.cc:1397] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:52.841637 14603 raft_consensus.cc:348] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.841805 14603 raft_consensus.cc:374] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:52.841848 14603 raft_consensus.cc:729] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c245c8b1a20245cdad4618feedff7cee, State: Initialized, Role: FOLLOWER
I20260620 14:33:52.841843 14602 raft_consensus.cc:348] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.841956 14603 consensus_queue.cc:260] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.842032 14602 raft_consensus.cc:374] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:52.842082 14602 raft_consensus.cc:729] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21451b18edca432291e22c3dac2ad168, State: Initialized, Role: FOLLOWER
I20260620 14:33:52.842254 14603 ts_tablet_manager.cc:1428] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:52.842217 14602 consensus_queue.cc:260] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.842386 14317 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
I20260620 14:33:52.842519 14451 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
I20260620 14:33:52.842595 14604 raft_consensus.cc:348] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.842756 14604 raft_consensus.cc:374] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:52.842799 14604 raft_consensus.cc:729] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d2319e31784891a6a991a62f87ac09, State: Initialized, Role: FOLLOWER
I20260620 14:33:52.842852 14602 ts_tablet_manager.cc:1428] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 14:33:52.842921 14604 consensus_queue.cc:260] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.843204 14604 ts_tablet_manager.cc:1428] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:52.843228 14585 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
W20260620 14:33:52.886523 14452 tablet.cc:2369] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:52.964710 14610 raft_consensus.cc:482] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:52.964843 14610 raft_consensus.cc:504] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.965270 14610 leader_election.cc:288] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731), c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
I20260620 14:33:52.968274 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.965560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37317 (local address 127.0.98.1:40175)
0620 14:33:52.965615 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.965618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.965845 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:33:52.965880 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.965880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.965893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.965953 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:52.966131 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.966610 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.967519 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.967906 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.967966 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.968001 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.968149 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.968197 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:33:52.968300 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.965551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33289 (local address 127.0.98.2:40731)
0620 14:33:52.965597 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:52.965600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:52.965639 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:33:52.965743 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:52.965744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:52.965759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:52.965832 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:52.966007 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.966554 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.967523 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.967915 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.968003 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.968014 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:52.968198 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:33:52.968239 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:33:52.968299 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.965518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.98.3:37317)
0620 14:33:52.965768 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.965778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.965834 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.965963 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.965964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.965965 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:52.966063 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.966065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.966699 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.966700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.967426 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.967428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.967998 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.967999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.968080 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.968084 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.968127 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.968178 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":68,"thread_start_us":101,"threads_started":1}
I20260620 14:33:52.968302 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:52.965517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40731 (local address 127.0.98.3:33289)
0620 14:33:52.965579 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:52.965592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:52.965658 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:52.965860 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:52.965861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:52.965863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:52.965937 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.965939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.966634 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.966635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.967423 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:52.967425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:52.968019 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:52.968020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:52.968102 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:52.968104 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:33:52.968180 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:33:52.968218 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":29}
I20260620 14:33:52.968724 14271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5be536edddf4904b653be612f74a3f3" candidate_uuid: "f3d2319e31784891a6a991a62f87ac09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c245c8b1a20245cdad4618feedff7cee" is_pre_election: true
I20260620 14:33:52.968859 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5be536edddf4904b653be612f74a3f3" candidate_uuid: "f3d2319e31784891a6a991a62f87ac09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21451b18edca432291e22c3dac2ad168" is_pre_election: true
I20260620 14:33:52.968928 14271 raft_consensus.cc:2454] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d2319e31784891a6a991a62f87ac09 in term 0.
I20260620 14:33:52.969027 14405 raft_consensus.cc:2454] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d2319e31784891a6a991a62f87ac09 in term 0.
I20260620 14:33:52.969170 14474 leader_election.cc:302] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee, f3d2319e31784891a6a991a62f87ac09; no voters:
I20260620 14:33:52.969357 14610 raft_consensus.cc:2781] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:52.969461 14610 raft_consensus.cc:482] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:52.969494 14610 raft_consensus.cc:3037] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:52.970171 14610 raft_consensus.cc:504] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.970320 14610 leader_election.cc:288] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [CANDIDATE]: Term 1 election: Requested vote from peers 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731), c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
I20260620 14:33:52.970477 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5be536edddf4904b653be612f74a3f3" candidate_uuid: "f3d2319e31784891a6a991a62f87ac09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21451b18edca432291e22c3dac2ad168"
I20260620 14:33:52.970492 14271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5be536edddf4904b653be612f74a3f3" candidate_uuid: "f3d2319e31784891a6a991a62f87ac09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c245c8b1a20245cdad4618feedff7cee"
I20260620 14:33:52.970592 14405 raft_consensus.cc:3037] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:52.970587 14271 raft_consensus.cc:3037] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:52.971390 14271 raft_consensus.cc:2454] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d2319e31784891a6a991a62f87ac09 in term 1.
I20260620 14:33:52.971405 14405 raft_consensus.cc:2454] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d2319e31784891a6a991a62f87ac09 in term 1.
I20260620 14:33:52.971570 14474 leader_election.cc:302] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee, f3d2319e31784891a6a991a62f87ac09; no voters:
I20260620 14:33:52.971683 14610 raft_consensus.cc:2781] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:52.971849 14610 raft_consensus.cc:686] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 1 LEADER]: Becoming Leader. State: Replica: f3d2319e31784891a6a991a62f87ac09, State: Running, Role: LEADER
I20260620 14:33:52.971969 14610 consensus_queue.cc:237] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:52.972877 14125 catalog_manager.cc:5495] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3d2319e31784891a6a991a62f87ac09 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f3d2319e31784891a6a991a62f87ac09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } health_report { overall_health: HEALTHY } } }
W20260620 14:33:52.980362 14318 tablet.cc:2369] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:53.013012 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.010796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.0.98.3:43019)
0620 14:33:53.010856 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.010859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.010867 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:33:53.010921 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.010922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.010942 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.011009 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:53.011132 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.011597 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.012302 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.012481 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.012533 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.012667 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:53.012766 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:53.012825 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.012855 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.012872 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:33:53.015017 14271 raft_consensus.cc:1261] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Refusing update from remote peer f3d2319e31784891a6a991a62f87ac09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:53.015030 14405 raft_consensus.cc:1261] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Refusing update from remote peer f3d2319e31784891a6a991a62f87ac09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:53.015410 14610 consensus_queue.cc:1035] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:53.015580 14612 consensus_queue.cc:1035] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:53.018617 14616 mvcc.cc:205] Tried to move back new op lower bound from 7298932871226339328 to 7298932871054000128. Current Snapshot: MvccSnapshot[applied={T|T < 7298932871226339328}]
I20260620 14:33:53.018959 14617 mvcc.cc:205] Tried to move back new op lower bound from 7298932871226339328 to 7298932871054000128. Current Snapshot: MvccSnapshot[applied={T|T < 7298932871226339328 or (T in {7298932871226339328})}]
I20260620 14:33:53.020315 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:53.020416 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:53.021185 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Sending DeleteTablet for 3 replicas of tablet a5be536edddf4904b653be612f74a3f3
I20260620 14:33:53.021549 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet a5be536edddf4904b653be612f74a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44800
I20260620 14:33:53.021552 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet a5be536edddf4904b653be612f74a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59414
I20260620 14:33:53.021803 14626 tablet_replica.cc:330] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: stopping tablet replica
I20260620 14:33:53.021803 14251 tablet_service.cc:1514] Processing DeleteTablet for tablet a5be536edddf4904b653be612f74a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33914
I20260620 14:33:53.021917 14626 raft_consensus.cc:2229] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:53.022032 14626 raft_consensus.cc:2258] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.022128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14188
I20260620 14:33:53.023393 14627 tablet_replica.cc:330] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: stopping tablet replica
I20260620 14:33:53.023504 14627 raft_consensus.cc:2229] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:53.023609 14627 raft_consensus.cc:2258] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.024065 14627 ts_tablet_manager.cc:1905] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:53.025753 14627 ts_tablet_manager.cc:1918] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:53.025830 14627 log.cc:1196] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/a5be536edddf4904b653be612f74a3f3
I20260620 14:33:53.026110 14627 ts_tablet_manager.cc:1939] T a5be536edddf4904b653be612f74a3f3 P f3d2319e31784891a6a991a62f87ac09: Deleting consensus metadata
I20260620 14:33:53.026538 14112 catalog_manager.cc:4841] TS f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): tablet a5be536edddf4904b653be612f74a3f3 (table test-table [id=1cada87998ff47c893e502df8ff58889]) successfully deleted
W20260620 14:33:53.027908 14112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40175 (error 108)
I20260620 14:33:53.028481 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.028338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.0.1:33916)
0620 14:33:53.028401 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.028444 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:33:53.028501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14322
I20260620 14:33:53.028681 14626 ts_tablet_manager.cc:1905] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:33:53.030174 14112 catalog_manager.cc:4584] TS c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5be536edddf4904b653be612f74a3f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40175: connect: Connection refused (error 111)
I20260620 14:33:53.035473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14456
I20260620 14:33:53.035597 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.035464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40731 (local address 127.0.0.1:44804)
0620 14:33:53.035529 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.035559 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:33:53.040081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14094
I20260620 14:33:53.045393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--webserver_interface=127.0.98.62
--webserver_port=34415
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:53.125962 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.126181 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.126205 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.127404 14630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:53.127447 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.127461 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.127501 14630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:53.127532 14630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:53.129040 14630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.129300 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.129566 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:53.138182 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.138217 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.138208 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.138403 14630 server_base.cc:1034] running on GCE node
I20260620 14:33:53.138650 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.138911 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.140080 14630 hybrid_clock.cc:648] HybridClock initialized: now 1781966033140068 us; error 27 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.142722 14630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:53.143198 14630 webserver.cc:460] Webserver started at http://127.0.98.62:34415/ using document root <none> and password file <none>
I20260620 14:33:53.143486 14630 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.143575 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.144953 14630 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:33:53.145645 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.145820 14630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:53.145929 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "80ef4d03b7d44720827181abb3c3e24f"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "6b393122ae0c85a4b59fbd5e457bcf44"
server_key_iv: "576066cdd0c0a074b3d3076570d0df5d"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.146335 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.155632 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.157913 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.158087 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.162055 14630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34007
I20260620 14:33:53.162103 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34007 every 8 connection(s)
I20260620 14:33:53.162503 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:53.164043 14699 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:53.164696 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap starting.
I20260620 14:33:53.169888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14630
I20260620 14:33:53.170183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40175
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=37893
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:53.173044 14699 log.cc:824] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.175096 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:53.175392 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap complete.
I20260620 14:33:53.177762 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.177990 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.178164 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.178227 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:53.178294 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:53.178347 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:33:53.179374 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.179461 14699 leader_election.cc:302] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ef4d03b7d44720827181abb3c3e24f; no voters:
I20260620 14:33:53.179682 14699 leader_election.cc:288] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:33:53.179716 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:33:53.179847 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 LEADER]: Becoming Leader. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Running, Role: LEADER
I20260620 14:33:53.179951 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:53.179955 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.180341 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ef4d03b7d44720827181abb3c3e24f. Latest consensus state: current_term: 2 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:53.180442 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:53.180346 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:53.180608 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:53.180801 14710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:53.181656 14710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:53.181768 14710 catalog_manager.cc:1259] Loaded cluster ID: 779e7d0890724e2ea4cc710c944f71ac
I20260620 14:33:53.181806 14710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:53.182057 14710 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:33:53.187335 14710 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:33:53.189898 14710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:53.190094 14710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Loaded TSK: 0
I20260620 14:33:53.190294 14710 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:33:53.255889 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.256122 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.256163 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.257310 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.257380 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.257424 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:33:53.258952 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37893
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.259224 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.259465 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:53.264519 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.264506 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.264530 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.265033 14701 server_base.cc:1034] running on GCE node
I20260620 14:33:53.265254 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.265550 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.266723 14701 hybrid_clock.cc:648] HybridClock initialized: now 1781966033266705 us; error 35 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.269199 14701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:33:53.269706 14701 webserver.cc:460] Webserver started at http://127.0.98.1:37893/ using document root <none> and password file <none>
I20260620 14:33:53.269937 14701 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.270023 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.271432 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.272181 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.272356 14701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.272455 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "c245c8b1a20245cdad4618feedff7cee"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "16fd332c9759b9afc5cc8e0376e0d2e8"
server_key_iv: "f5f24807534e260e75954bdafe0b31bc"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.272833 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.281118 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.283553 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.283720 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.283960 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:53.284400 14743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:53.286377 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:53.286439 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:33:53.286487 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:53.287159 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:53.287223 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:53.287295 14743 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Bootstrap starting.
I20260620 14:33:53.294440 14701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40175
I20260620 14:33:53.294477 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40175 every 8 connection(s)
I20260620 14:33:53.294878 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:53.295238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14701
I20260620 14:33:53.295557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40731
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=36493
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:53.298168 14743 log.cc:824] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.300565 14743 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:53.300890 14743 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Bootstrap complete.
I20260620 14:33:53.302109 14743 ts_tablet_manager.cc:1397] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.014s
I20260620 14:33:53.303522 14743 raft_consensus.cc:348] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:53.303835 14743 raft_consensus.cc:729] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c245c8b1a20245cdad4618feedff7cee, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.303962 14743 consensus_queue.cc:260] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:53.304250 14743 ts_tablet_manager.cc:1428] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:53.311723 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.295725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39961 (local address 127.0.98.62:34007)
0620 14:33:53.299951 (+ 4226us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.299962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.300731 (+ 769us) server_negotiation.cc:407] Connection header received
0620 14:33:53.301275 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.301281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.301341 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.301480 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:53.302528 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.304161 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.305224 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.305856 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.305993 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.308189 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:53.308194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:53.308194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:53.308206 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:53.310032 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.310447 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.310450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.310451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.310549 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.310803 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.310804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.310804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.311025 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:53.311114 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.311404 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.311532 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4163,"thread_start_us":119,"threads_started":1}
I20260620 14:33:53.311736 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.295614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:39961)
0620 14:33:53.300106 (+ 4492us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.300156 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.300996 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.301615 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.301619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.302049 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:53.302393 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.302399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.304265 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.304266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.305113 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.305115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.305967 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.305968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.306098 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.306677 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:53.306686 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:53.308064 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:53.310123 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.310129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.310132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.310350 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.310628 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.310629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.310629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.310735 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.311105 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:53.311111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:53.311244 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.311350 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.311524 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4448,"thread_start_us":92,"threads_started":1}
I20260620 14:33:53.312487 14851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:53.312615 14851 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:53.312789 14851 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:53.313572 14663 ts_manager.cc:194] Registered new tserver with Master: c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
I20260620 14:33:53.314667 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39961
I20260620 14:33:53.315490 14851 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
I20260620 14:33:53.323403 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.314151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.0.1:33930)
0620 14:33:53.320148 (+ 5997us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.320183 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.320240 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.320497 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.320500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.320502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:53.320581 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.320585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.321536 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.321537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.322355 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.322357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.323070 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.323072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.323161 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.323173 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.323271 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.323327 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5966,"thread_start_us":84,"threads_started":1}
I20260620 14:33:53.323413 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.314380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.98.1:40175)
0620 14:33:53.320345 (+ 5965us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.320348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.320356 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:33:53.320382 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.320384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.320395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.320480 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:53.320661 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.321393 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.322471 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.322944 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.323056 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.323116 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.323288 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.323327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5939,"thread_start_us":73,"threads_started":1}
I20260620 14:33:53.324133 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet a5be536edddf4904b653be612f74a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33930
I20260620 14:33:53.324726 14860 tablet_replica.cc:330] stopping tablet replica
I20260620 14:33:53.324836 14860 raft_consensus.cc:2229] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:53.324905 14860 raft_consensus.cc:2258] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.325207 14860 ts_tablet_manager.cc:1905] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:53.327165 14860 ts_tablet_manager.cc:1918] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:53.327226 14860 log.cc:1196] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/a5be536edddf4904b653be612f74a3f3
I20260620 14:33:53.327457 14860 ts_tablet_manager.cc:1939] T a5be536edddf4904b653be612f74a3f3 P c245c8b1a20245cdad4618feedff7cee: Deleting consensus metadata
I20260620 14:33:53.327981 14649 catalog_manager.cc:4841] TS c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175): tablet a5be536edddf4904b653be612f74a3f3 (table test-table [id=1cada87998ff47c893e502df8ff58889]) successfully deleted
W20260620 14:33:53.386912 14854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.387179 14854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.387226 14854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.388415 14854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.388484 14854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.388530 14854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:33:53.390092 14854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36493
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.390386 14854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.390630 14854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:53.395684 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.395706 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.395684 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.395982 14854 server_base.cc:1034] running on GCE node
I20260620 14:33:53.396237 14854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.396497 14854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.397639 14854 hybrid_clock.cc:648] HybridClock initialized: now 1781966033397629 us; error 32 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.400174 14854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:33:53.400630 14854 webserver.cc:460] Webserver started at http://127.0.98.2:36493/ using document root <none> and password file <none>
I20260620 14:33:53.400869 14854 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.400955 14854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.402352 14854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.403057 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.403231 14854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:53.403332 14854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "21451b18edca432291e22c3dac2ad168"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "3c2e8c213e7f40cd3dc22c7dd9b4e74c"
server_key_iv: "c62112a63a134b390c4fe0f099890de7"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.403678 14854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.429195 14854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.431253 14854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.431423 14854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.431672 14854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:53.432121 14882 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:53.433619 14854 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:53.433678 14854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:33:53.433743 14854 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:53.434373 14854 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:53.434430 14854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:53.434485 14882 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Bootstrap starting.
I20260620 14:33:53.435993 14882 log.cc:824] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.439594 14882 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:53.439937 14882 tablet_bootstrap.cc:492] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Bootstrap complete.
I20260620 14:33:53.440115 14882 ts_tablet_manager.cc:1397] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260620 14:33:53.441310 14854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40731
I20260620 14:33:53.441387 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40731 every 8 connection(s)
I20260620 14:33:53.441720 14854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:53.442404 14882 raft_consensus.cc:348] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:53.442802 14882 raft_consensus.cc:729] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21451b18edca432291e22c3dac2ad168, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.442946 14882 consensus_queue.cc:260] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } }
I20260620 14:33:53.443214 14882 ts_tablet_manager.cc:1428] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Time spent starting tablet: real 0.003s user 0.003s sys 0.003s
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:53.451306 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.442536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34713 (local address 127.0.98.62:34007)
0620 14:33:53.442597 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.442599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.442782 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:33:53.442949 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.442950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.442966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.443041 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:53.443788 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.444407 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.445635 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.445940 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.446009 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.448554 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:53.448555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:53.448556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:53.448563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:53.450179 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.450536 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.450537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.450537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.450588 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.450796 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.450797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.450797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.450898 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:53.450946 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.451174 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.451240 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:33:53.451514 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.442493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:34713)
0620 14:33:53.442663 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.442697 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.442871 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.443100 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.443104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.443469 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:53.443675 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.443681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.444530 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.444532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.445556 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.445559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.446045 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.446046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.446229 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.446739 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:53.446748 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:53.448461 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:53.450247 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.450252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.450254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.450462 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.450646 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.450646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.450647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.450743 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.450950 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:53.450954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:53.451067 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.451149 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.451273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260620 14:33:53.451815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14854
I20260620 14:33:53.452024 14991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:53.452121 14991 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:53.452127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43019
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=43003
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:53.452323 14991 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:53.452899 14663 ts_manager.cc:194] Registered new tserver with Master: 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
I20260620 14:33:53.453531 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34713
I20260620 14:33:53.454424 14991 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:33:53.461711 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.453203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40731 (local address 127.0.0.1:44818)
0620 14:33:53.453256 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.453266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.453306 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.453702 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.453703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.454022 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:53.454056 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.454072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.455190 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.455191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.455689 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.455690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.456190 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.456193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.456257 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.456750 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:53.456754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:53.458331 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:53.460501 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.460508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.460510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.460709 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.460976 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.460976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.460977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.461049 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.461412 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:53.461453 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:53.461507 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.461564 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.461644 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:33:53.461886 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.453261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.98.2:40731)
0620 14:33:53.453504 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.453509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.453529 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:33:53.453569 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.453571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.453587 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.453689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:53.454160 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.455096 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.455773 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.456101 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.456181 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.458426 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:53.458435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:53.458435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:53.458444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:53.460411 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.460819 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.460823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.460824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.460893 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.461129 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.461130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.461130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.461331 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:53.461460 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.461627 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.461740 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260620 14:33:53.462306 14925 tablet_service.cc:1514] Processing DeleteTablet for tablet a5be536edddf4904b653be612f74a3f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44818
I20260620 14:33:53.462904 14997 tablet_replica.cc:330] stopping tablet replica
I20260620 14:33:53.463032 14997 raft_consensus.cc:2229] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:53.463119 14997 raft_consensus.cc:2258] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.463469 14997 ts_tablet_manager.cc:1905] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:53.465495 14997 ts_tablet_manager.cc:1918] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:53.465591 14997 log.cc:1196] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/a5be536edddf4904b653be612f74a3f3
I20260620 14:33:53.465808 14997 ts_tablet_manager.cc:1939] T a5be536edddf4904b653be612f74a3f3 P 21451b18edca432291e22c3dac2ad168: Deleting consensus metadata
I20260620 14:33:53.466279 14647 catalog_manager.cc:4841] TS 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): tablet a5be536edddf4904b653be612f74a3f3 (table test-table [id=1cada87998ff47c893e502df8ff58889]) successfully deleted
W20260620 14:33:53.537906 14995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.538137 14995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.538175 14995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.539319 14995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.539392 14995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.539440 14995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:33:53.540915 14995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43003
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.541206 14995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.541457 14995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:53.546597 15005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.546689 14995 server_base.cc:1034] running on GCE node
W20260620 14:33:53.546597 15003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.546633 15002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.547062 14995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.547262 14995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.548437 14995 hybrid_clock.cc:648] HybridClock initialized: now 1781966033548426 us; error 31 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.550951 14995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:33:53.551330 14995 webserver.cc:460] Webserver started at http://127.0.98.3:43003/ using document root <none> and password file <none>
I20260620 14:33:53.551515 14995 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.551569 14995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.552841 14995 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.553390 15012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.553557 14995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:53.553661 14995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "f3d2319e31784891a6a991a62f87ac09"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "9d13bd21896f54c62e5c669766b06a89"
server_key_iv: "237dfa1168d8f2b6201cd3d4dc5ab6c7"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.554003 14995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.570912 14995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.572791 14995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.572964 14995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.573206 14995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:53.573573 14995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:53.573628 14995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.573669 14995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:53.573696 14995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.579579 14995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43019
I20260620 14:33:53.579638 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43019 every 8 connection(s)
I20260620 14:33:53.580003 14995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:33:53.587486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 14995
I20260620 14:33:53.589541 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.580742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45921 (local address 127.0.98.62:34007)
0620 14:33:53.580808 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.580810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.581007 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:33:53.581689 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.581691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.581705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.581759 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:53.582708 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.583209 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.584211 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.584418 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.584499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.586616 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:53.586618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:53.586618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:53.586626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:53.588105 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.588518 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.588520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.588520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.588597 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.588837 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.588838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.588839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.588970 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:53.589020 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.589337 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.589403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:33:53.589711 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.580734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:45921)
0620 14:33:53.580890 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.580925 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.581581 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.581828 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.581832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.582403 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:53.582604 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.582609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.583303 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.583304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.584114 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.584116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.584507 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.584508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.584658 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.585186 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:53.585195 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:53.586514 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:53.588177 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.588185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.588187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.588444 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.588670 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:53.588670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:53.588671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:53.588769 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:53.589044 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:53.589049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:53.589201 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.589314 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.589492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 14:33:53.590219 15126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34007
I20260620 14:33:53.590309 15126 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:53.590499 15126 heartbeater.cc:511] Master 127.0.98.62:34007 requested a full tablet report, sending...
I20260620 14:33:53.590852 14663 ts_manager.cc:194] Registered new tserver with Master: f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019)
I20260620 14:33:53.591221 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45921
I20260620 14:33:53.594699 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.587891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.0.98.62:34007)
0620 14:33:53.588000 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.588002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.588055 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:33:53.588133 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.588133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.588162 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.588238 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:53.588882 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.589615 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.590809 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.591075 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.591151 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.592451 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:53.592453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:53.592453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:53.592472 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:53.593802 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.594100 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.594100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.594101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.594148 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:53.594304 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:53.594305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:53.594305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:53.594396 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:53.594519 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.594548 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.594607 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260620 14:33:53.595397 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:53.597837 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.595721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.0.98.62:34007)
0620 14:33:53.595773 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.595774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.595844 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:33:53.595895 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.595896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.595923 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.595980 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:53.596066 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.596498 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.597180 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.597359 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.597408 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.597591 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:53.597680 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:53.597742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.597772 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.597791 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:33:53.598093 14663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:53.599696 14663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:53.601809 14925 tablet_service.cc:1467] Processing CreateTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e (DEFAULT_TABLE table=test-table [id=030cac2f32a04a7eabec421742db1481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:53.602015 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e038aa58f9dd4dff8d981b3da6f9e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:53.602068 14784 tablet_service.cc:1467] Processing CreateTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e (DEFAULT_TABLE table=test-table [id=030cac2f32a04a7eabec421742db1481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:53.602244 14784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e038aa58f9dd4dff8d981b3da6f9e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:53.603629 14743 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Bootstrap starting.
I20260620 14:33:53.603695 14882 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Bootstrap starting.
I20260620 14:33:53.604081 14882 tablet_bootstrap.cc:654] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:53.604094 14743 tablet_bootstrap.cc:654] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:53.604849 14882 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: No bootstrap required, opened a new log
I20260620 14:33:53.604903 14882 ts_tablet_manager.cc:1397] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:53.605044 14743 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: No bootstrap required, opened a new log
I20260620 14:33:53.605103 14743 ts_tablet_manager.cc:1397] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:33:53.605221 14882 raft_consensus.cc:348] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.605257 14743 raft_consensus.cc:348] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.605298 14882 raft_consensus.cc:374] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:53.605345 14882 raft_consensus.cc:729] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21451b18edca432291e22c3dac2ad168, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.605329 14743 raft_consensus.cc:374] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:53.605403 14743 raft_consensus.cc:729] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c245c8b1a20245cdad4618feedff7cee, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.605397 14882 consensus_queue.cc:260] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.605469 14743 consensus_queue.cc:260] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.605533 14882 ts_tablet_manager.cc:1428] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.605543 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.601588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43019 (local address 127.0.0.1:59442)
0620 14:33:53.601641 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.601653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.601695 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.602036 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.602037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.602039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:53.602081 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.602083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.603359 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.603360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.604302 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.604305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.605246 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.605247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.605317 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.605321 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.605393 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.605475 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:33:53.605572 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.601691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.98.3:43019)
0620 14:33:53.601866 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.601871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.601882 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:33:53.601920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.601922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.601939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.602025 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:53.602162 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.603235 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.604400 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.605125 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.605237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.605302 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.605440 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.605489 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260620 14:33:53.605572 14743 ts_tablet_manager.cc:1428] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:33:53.606601 15060 tablet_service.cc:1467] Processing CreateTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e (DEFAULT_TABLE table=test-table [id=030cac2f32a04a7eabec421742db1481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:53.606946 15060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e038aa58f9dd4dff8d981b3da6f9e75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:53.609764 15135 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Bootstrap starting.
I20260620 14:33:53.610332 15135 tablet_bootstrap.cc:654] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:53.610795 15135 log.cc:824] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.611464 15135 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: No bootstrap required, opened a new log
I20260620 14:33:53.611552 15135 ts_tablet_manager.cc:1397] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:33:53.612901 15135 raft_consensus.cc:348] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.613039 15135 raft_consensus.cc:374] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:53.613062 15135 raft_consensus.cc:729] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d2319e31784891a6a991a62f87ac09, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.613248 15135 consensus_queue.cc:260] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.613550 15135 ts_tablet_manager.cc:1428] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:33:53.613596 15126 heartbeater.cc:503] Master 127.0.98.62:34007 was elected leader, sending a full tablet report...
I20260620 14:33:53.638939 14994 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:53.639135 14994 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.639541 14994 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
I20260620 14:33:53.642732 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.639866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.98.2:50701)
0620 14:33:53.640084 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.640095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.640152 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.640291 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.640291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.640293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:53.640378 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.640380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.641017 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.641018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.641856 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.641858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.642432 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.642434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.642510 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.642513 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.642596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.642635 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260620 14:33:53.642724 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.640085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50701 (local address 127.0.98.1:40175)
0620 14:33:53.640127 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.640128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.640136 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:33:53.640218 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.640218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.640230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.640287 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:53.640463 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.640929 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.641962 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.642334 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.642420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.642432 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.642605 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.642653 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:33:53.642859 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.639739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43019 (local address 127.0.98.2:52711)
0620 14:33:53.639826 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.639840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.639905 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:53.640137 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:53.640138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:53.640141 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:53.640260 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.640263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.640973 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.640974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.641820 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:53.641822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.642385 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.642386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:53.642474 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.642492 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:33:53.642721 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 14:33:53.642767 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:33:53.643052 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.639812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52711 (local address 127.0.98.3:43019)
0620 14:33:53.639885 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.639887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.639896 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:53.639980 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.639981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.639993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.640355 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:53.640411 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.640886 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.641910 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.642291 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.642385 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.642395 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.642733 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.642980 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:33:53.643222 14805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e038aa58f9dd4dff8d981b3da6f9e75e" candidate_uuid: "21451b18edca432291e22c3dac2ad168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c245c8b1a20245cdad4618feedff7cee" is_pre_election: true
I20260620 14:33:53.643317 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e038aa58f9dd4dff8d981b3da6f9e75e" candidate_uuid: "21451b18edca432291e22c3dac2ad168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d2319e31784891a6a991a62f87ac09" is_pre_election: true
I20260620 14:33:53.643415 14805 raft_consensus.cc:2454] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21451b18edca432291e22c3dac2ad168 in term 0.
I20260620 14:33:53.643496 15080 raft_consensus.cc:2454] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21451b18edca432291e22c3dac2ad168 in term 0.
I20260620 14:33:53.643651 14878 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21451b18edca432291e22c3dac2ad168, c245c8b1a20245cdad4618feedff7cee; no voters:
I20260620 14:33:53.643771 14994 raft_consensus.cc:2781] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:53.643839 14994 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:53.643873 14994 raft_consensus.cc:3037] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:53.644701 14994 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.644841 14994 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [CANDIDATE]: Term 1 election: Requested vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175)
I20260620 14:33:53.645056 14805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e038aa58f9dd4dff8d981b3da6f9e75e" candidate_uuid: "21451b18edca432291e22c3dac2ad168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c245c8b1a20245cdad4618feedff7cee"
I20260620 14:33:53.645067 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e038aa58f9dd4dff8d981b3da6f9e75e" candidate_uuid: "21451b18edca432291e22c3dac2ad168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d2319e31784891a6a991a62f87ac09"
I20260620 14:33:53.645134 14805 raft_consensus.cc:3037] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:53.645157 15080 raft_consensus.cc:3037] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:53.645978 14805 raft_consensus.cc:2454] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21451b18edca432291e22c3dac2ad168 in term 1.
I20260620 14:33:53.645989 15080 raft_consensus.cc:2454] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21451b18edca432291e22c3dac2ad168 in term 1.
I20260620 14:33:53.646143 14878 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09; no voters:
I20260620 14:33:53.646253 14994 raft_consensus.cc:2781] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:53.646458 14994 raft_consensus.cc:686] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 1 LEADER]: Becoming Leader. State: Replica: 21451b18edca432291e22c3dac2ad168, State: Running, Role: LEADER
I20260620 14:33:53.646544 14994 consensus_queue.cc:237] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.647296 14662 catalog_manager.cc:5495] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21451b18edca432291e22c3dac2ad168 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "21451b18edca432291e22c3dac2ad168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } health_report { overall_health: HEALTHY } } }
I20260620 14:33:53.653203 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.651187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.98.2:40731)
0620 14:33:53.651229 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:53.651232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:53.651241 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:53.651298 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:53.651298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:53.651313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:53.651378 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:53.651464 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.651938 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.652582 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:53.652779 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:53.652829 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:53.652950 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:53.653038 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:53.653088 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:53.653141 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:33:53.653155 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:33:53.654639 15080 raft_consensus.cc:1261] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Refusing update from remote peer 21451b18edca432291e22c3dac2ad168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:53.654670 14805 raft_consensus.cc:1261] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Refusing update from remote peer 21451b18edca432291e22c3dac2ad168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:53.654987 15139 consensus_queue.cc:1035] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:53.655093 14994 consensus_queue.cc:1035] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:53.657227 15143 mvcc.cc:205] Tried to move back new op lower bound from 7298932873847029760 to 7298932873816825856. Current Snapshot: MvccSnapshot[applied={T|T < 7298932873847029760}]
I20260620 14:33:53.659449 14662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:53.659538 14662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:53.660254 14662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Sending DeleteTablet for 3 replicas of tablet e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:33:53.660559 14925 tablet_service.cc:1514] Processing DeleteTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44818
I20260620 14:33:53.660655 14997 tablet_replica.cc:330] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: stopping tablet replica
I20260620 14:33:53.660627 15060 tablet_service.cc:1514] Processing DeleteTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59442
I20260620 14:33:53.660704 14997 raft_consensus.cc:2229] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:53.660789 14997 raft_consensus.cc:2258] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.660787 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet e038aa58f9dd4dff8d981b3da6f9e75e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33930
I20260620 14:33:53.660883 14860 tablet_replica.cc:330] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: stopping tablet replica
I20260620 14:33:53.660902 15151 tablet_replica.cc:330] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: stopping tablet replica
I20260620 14:33:53.660949 14860 raft_consensus.cc:2229] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:53.660988 15151 raft_consensus.cc:2229] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:53.661006 14860 raft_consensus.cc:2258] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.661092 15151 raft_consensus.cc:2258] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:53.661365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14701
I20260620 14:33:53.661379 14997 ts_tablet_manager.cc:1905] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:53.662956 14997 ts_tablet_manager.cc:1918] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:53.663007 14997 log.cc:1196] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:33:53.663235 14997 ts_tablet_manager.cc:1939] T e038aa58f9dd4dff8d981b3da6f9e75e P 21451b18edca432291e22c3dac2ad168: Deleting consensus metadata
I20260620 14:33:53.663563 14647 catalog_manager.cc:4841] TS 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): tablet e038aa58f9dd4dff8d981b3da6f9e75e (table test-table [id=030cac2f32a04a7eabec421742db1481]) successfully deleted
I20260620 14:33:53.664166 15151 ts_tablet_manager.cc:1905] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:53.665300 15151 ts_tablet_manager.cc:1918] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:53.665346 15151 log.cc:1196] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:33:53.665654 15151 ts_tablet_manager.cc:1939] T e038aa58f9dd4dff8d981b3da6f9e75e P f3d2319e31784891a6a991a62f87ac09: Deleting consensus metadata
I20260620 14:33:53.666050 14649 catalog_manager.cc:4841] TS f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): tablet e038aa58f9dd4dff8d981b3da6f9e75e (table test-table [id=030cac2f32a04a7eabec421742db1481]) successfully deleted
W20260620 14:33:53.667572 14649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40175 (error 108)
I20260620 14:33:53.667806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14854
I20260620 14:33:53.668433 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:53.667885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40175 (local address 127.0.0.1:33934)
0620 14:33:53.668360 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.668396 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
W20260620 14:33:53.669150 14649 catalog_manager.cc:4584] TS c245c8b1a20245cdad4618feedff7cee (127.0.98.1:40175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e038aa58f9dd4dff8d981b3da6f9e75e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40175: connect: Connection refused (error 111)
I20260620 14:33:53.673015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14995
I20260620 14:33:53.677910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 14630
I20260620 14:33:53.683205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--webserver_interface=127.0.98.62
--webserver_port=34415
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:53.766623 15153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.766840 15153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.766873 15153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.768062 15153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:53.768103 15153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.768117 15153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.768129 15153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:53.768184 15153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:53.769619 15153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.769872 15153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.770129 15153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:53.774893 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.774899 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.774893 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.775193 15153 server_base.cc:1034] running on GCE node
I20260620 14:33:53.775431 15153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.775713 15153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.776901 15153 hybrid_clock.cc:648] HybridClock initialized: now 1781966033776885 us; error 47 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.779385 15153 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:33:53.779874 15153 webserver.cc:460] Webserver started at http://127.0.98.62:34415/ using document root <none> and password file <none>
I20260620 14:33:53.780123 15153 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.780194 15153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.781550 15153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.782367 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.782533 15153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.782646 15153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "80ef4d03b7d44720827181abb3c3e24f"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "6b393122ae0c85a4b59fbd5e457bcf44"
server_key_iv: "576066cdd0c0a074b3d3076570d0df5d"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.783011 15153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.806229 15153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.808099 15153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.808267 15153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.812405 15153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34007
I20260620 14:33:53.812439 15220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34007 every 8 connection(s)
I20260620 14:33:53.812811 15153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:53.814371 15221 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:53.814947 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap starting.
I20260620 14:33:53.818082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15153
I20260620 14:33:53.818349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40175
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=37893
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:53.823305 15221 log.cc:824] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.825713 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:53.826020 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f: Bootstrap complete.
I20260620 14:33:53.828275 15221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.828534 15221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.828718 15221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.828805 15221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:53.828857 15221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:53.828908 15221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:33:53.829985 15221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.830106 15221 leader_election.cc:302] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ef4d03b7d44720827181abb3c3e24f; no voters:
I20260620 14:33:53.830302 15221 leader_election.cc:288] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:33:53.830338 15225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:33:53.830454 15225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [term 3 LEADER]: Becoming Leader. State: Replica: 80ef4d03b7d44720827181abb3c3e24f, State: Running, Role: LEADER
I20260620 14:33:53.830542 15225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } }
I20260620 14:33:53.830592 15221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:53.830783 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ef4d03b7d44720827181abb3c3e24f. Latest consensus state: current_term: 3 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:53.830857 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:53.830789 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "80ef4d03b7d44720827181abb3c3e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ef4d03b7d44720827181abb3c3e24f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34007 } } }
I20260620 14:33:53.831025 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ef4d03b7d44720827181abb3c3e24f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:53.831525 15229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:53.832350 15229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:53.832450 15229 catalog_manager.cc:1259] Loaded cluster ID: 779e7d0890724e2ea4cc710c944f71ac
I20260620 14:33:53.832486 15229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:33:53.832703 15229 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9782d879ad google::LogMessage::Fail() at ??:0
@ 0x7f9782d8a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9782d874de google::LogMessage::Flush() at ??:0
@ 0x7f9782d88a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f978aca80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f978acb18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9784c550d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9784c4a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9785aae6db start_thread at ??:0
@ 0x7f978188561f clone at ??:0
W20260620 14:33:53.912714 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:53.912945 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:53.912984 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:53.914252 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:53.914321 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:53.914368 15223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:33:53.916074 15223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37893
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:53.916359 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:53.916592 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:53.921677 15223 server_base.cc:1034] running on GCE node
W20260620 14:33:53.921780 15260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.921655 15257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:53.921833 15258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:53.922139 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:53.922423 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:53.923599 15223 hybrid_clock.cc:648] HybridClock initialized: now 1781966033923576 us; error 42 us; skew 500 ppm
Jun 20 14:33:53 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:53.926121 15223 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:33:53.926549 15223 webserver.cc:460] Webserver started at http://127.0.98.1:37893/ using document root <none> and password file <none>
I20260620 14:33:53.926800 15223 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:53.926870 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:53.928264 15223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.929064 15267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:53.929368 15223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:53.929509 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "c245c8b1a20245cdad4618feedff7cee"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "16fd332c9759b9afc5cc8e0376e0d2e8"
server_key_iv: "f5f24807534e260e75954bdafe0b31bc"
server_key_version: "encryptionkey@0"
I20260620 14:33:53.929876 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:53.945261 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:53.947523 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:53.947685 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:53.947929 15223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:53.948372 15274 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:53.950572 15223 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:53.950639 15223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:33:53.950688 15223 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:53.951332 15223 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:53.951388 15223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:53.951524 15274 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Bootstrap starting.
I20260620 14:33:53.958621 15223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40175
I20260620 14:33:53.958752 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40175 every 8 connection(s)
I20260620 14:33:53.959043 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:53.962857 15274 log.cc:824] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:53.963411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15223
I20260620 14:33:53.963722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40731
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=36493
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:53.965719 15274 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:53.966042 15274 tablet_bootstrap.cc:492] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Bootstrap complete.
I20260620 14:33:53.967286 15274 ts_tablet_manager.cc:1397] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260620 14:33:53.968822 15274 raft_consensus.cc:348] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.969153 15274 raft_consensus.cc:729] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c245c8b1a20245cdad4618feedff7cee, State: Initialized, Role: FOLLOWER
I20260620 14:33:53.969326 15274 consensus_queue.cc:260] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:53.969650 15274 ts_tablet_manager.cc:1428] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260620 14:33:54.047330 15384 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:33:53.959693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:46695)
0620 14:33:53.959859 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:53.959902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:53.960399 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:54.047165 (+ 86766us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260620 14:33:54.047616 15382 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34007 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34007: Client connection negotiation failed: client connection to 127.0.98.62:34007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:33:54.047959 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.047844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:45429)
0620 14:33:54.047896 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.047938 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:33:54.048280 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.048181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:37447)
0620 14:33:54.048225 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.048249 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:33:54.048378 15382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:54.049180 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:54.049448 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:54.049505 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:54.050733 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:54.050799 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:54.050848 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:33:54.052366 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36493
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:54.052639 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:54.052865 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:54.057843 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:54.057880 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:54.057888 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:54.058177 15386 server_base.cc:1034] running on GCE node
I20260620 14:33:54.058401 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:54.058634 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:54.059779 15386 hybrid_clock.cc:648] HybridClock initialized: now 1781966034059765 us; error 28 us; skew 500 ppm
Jun 20 14:33:54 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:54.062191 15386 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:33:54.062587 15386 webserver.cc:460] Webserver started at http://127.0.98.2:36493/ using document root <none> and password file <none>
I20260620 14:33:54.062767 15386 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:54.062808 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:54.064059 15386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:54.064620 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.064777 15386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:54.064864 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "21451b18edca432291e22c3dac2ad168"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "3c2e8c213e7f40cd3dc22c7dd9b4e74c"
server_key_iv: "c62112a63a134b390c4fe0f099890de7"
server_key_version: "encryptionkey@0"
I20260620 14:33:54.065217 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:54.074405 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:54.076211 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:54.076321 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:54.076506 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:54.076748 15386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:54.076778 15386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.076799 15386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:54.076812 15386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.082654 15386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40731
I20260620 14:33:54.082722 15515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40731 every 8 connection(s)
I20260620 14:33:54.083072 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:54.084292 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.083921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:47575)
0620 14:33:54.084078 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.084151 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260620 14:33:54.084714 15516 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34007 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34007: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
I20260620 14:33:54.085049 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.084944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:50311)
0620 14:33:54.085001 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.085030 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:33:54.085352 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.085266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:55471)
0620 14:33:54.085303 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.085326 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:33:54.085508 15516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:33:54.088850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15386
I20260620 14:33:54.089116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43019
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34007
--webserver_port=43003
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:54.170032 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:54.170269 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:54.170306 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:54.171468 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:54.171536 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:54.171582 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:33:54.173058 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43003
--tserver_master_addrs=127.0.98.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:33:54.173324 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:54.173607 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:54.178435 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:54.178507 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:54.178463 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:54.178808 15519 server_base.cc:1034] running on GCE node
I20260620 14:33:54.179040 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:54.179287 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:54.180476 15519 hybrid_clock.cc:648] HybridClock initialized: now 1781966034180469 us; error 44 us; skew 500 ppm
Jun 20 14:33:54 dist-test-slave-tc2g krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:54.183094 15519 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:33:54.183575 15519 webserver.cc:460] Webserver started at http://127.0.98.3:43003/ using document root <none> and password file <none>
I20260620 14:33:54.183772 15519 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:54.183813 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:54.185111 15519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:54.185686 15534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.185880 15519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:33:54.185951 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "f3d2319e31784891a6a991a62f87ac09"
format_stamp: "Formatted at 2026-06-20 14:33:52 on dist-test-slave-tc2g"
server_key: "9d13bd21896f54c62e5c669766b06a89"
server_key_iv: "237dfa1168d8f2b6201cd3d4dc5ab6c7"
server_key_version: "encryptionkey@0"
I20260620 14:33:54.186267 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:54.194207 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:54.196448 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:54.196565 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:54.196743 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:54.196991 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:54.197021 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.197052 15519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:54.197067 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:54.202991 15519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43019
I20260620 14:33:54.203027 15647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43019 every 8 connection(s)
I20260620 14:33:54.203420 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:54.203756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15519
I20260620 14:33:54.204540 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.204135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:41701)
0620 14:33:54.204315 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.204403 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
W20260620 14:33:54.204912 15648 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34007 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34007: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
I20260620 14:33:54.205199 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.205099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:47683)
0620 14:33:54.205150 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.205180 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:33:54.205516 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:54.205401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:54779)
0620 14:33:54.205471 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:54.205496 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:33:54.205647 15648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:33:55.048918 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.048699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:51991)
0620 14:33:55.048818 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:55.048855 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260620 14:33:55.086251 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.085969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:41311)
0620 14:33:55.086116 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:55.086182 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260620 14:33:55.206137 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.205935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:38473)
0620 14:33:55.206043 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:55.206075 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260620 14:33:55.740082 15655 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:55.740202 15655 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:55.740588 15655 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
Jun 20 14:33:55 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:33:55 dist-test-slave-tc2g krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:33:55.751366 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.740849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45983 (local address 127.0.98.3:43019)
0620 14:33:55.741083 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:55.741092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:55.741111 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:33:55.741594 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:55.741596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:55.741641 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:55.741803 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:55.742526 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.743381 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744385 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.744782 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744905 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:55.747273 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:55.747277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:55.747277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:55.747287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:55.749359 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:55.749953 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:55.749957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:55.749957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:55.750065 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:55.750373 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:55.750374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:55.750374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:55.750678 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:55.750764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:55.751116 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:33:55.751255 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
I20260620 14:33:55.751366 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.740884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44077 (local address 127.0.98.2:40731)
0620 14:33:55.741062 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:55.741079 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:55.741097 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:33:55.741594 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:55.741596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:55.741641 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:55.741804 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:55.742526 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.743392 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744387 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.744786 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:55.747675 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:55.747679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:55.747679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:55.747689 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:55.749555 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:55.749953 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:55.749957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:55.749957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:55.750065 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:55.750373 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:55.750374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:55.750374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:55.750677 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:55.750764 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:55.751116 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:33:55.751254 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":53,"threads_started":1}
I20260620 14:33:55.751360 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.740741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40731 (local address 127.0.98.1:44077)
0620 14:33:55.740878 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:55.740892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:55.740965 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:55.741840 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:55.741842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:55.742215 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:55.742411 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:55.742416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.743501 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.743502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:55.744278 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:55.744281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.744885 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:55.744997 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:55.745643 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:55.745651 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:55.747556 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:55.749620 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:55.749621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:55.749621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:55.749866 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:55.750142 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:55.750143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:55.750143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:55.750297 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:55.750756 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:55.750762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:55.750911 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:33:55.751044 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:33:55.751244 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":81,"thread_start_us":72,"threads_started":1}
I20260620 14:33:55.751411 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:55.740741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43019 (local address 127.0.98.1:45983)
0620 14:33:55.740887 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:55.740898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:55.740965 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:55.741840 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:55.741842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:55.742216 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:55.742411 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:55.742416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.743489 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.743489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:55.744280 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:55.744282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:55.744883 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:55.744885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:55.744999 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:55.745692 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:55.745696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:55.747169 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:55.749483 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:55.749489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:55.749491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:55.749866 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:55.750142 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:55.750143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:55.750143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:55.750297 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:55.750756 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:55.750762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:55.750911 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:33:55.751044 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:33:55.751296 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":144,"thread_start_us":62,"threads_started":1}
W20260620 14:33:55.752243 15268 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
W20260620 14:33:55.752861 15270 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:33:55.752911 15270 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee; no voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09
I20260620 14:33:55.753031 15655 raft_consensus.cc:2726] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:56.049667 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:56.049522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:43873)
0620 14:33:56.049591 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:56.049628 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:33:56.086902 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:56.086683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:50299)
0620 14:33:56.086815 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:56.086857 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260620 14:33:56.206828 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:56.206586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:55003)
0620 14:33:56.206728 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:56.206778 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260620 14:33:57.050457 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:57.050229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:36497)
0620 14:33:57.050370 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:57.050410 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:33:57.087574 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:57.087367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:53935)
0620 14:33:57.087492 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:57.087531 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260620 14:33:57.207613 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:57.207350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:41779)
0620 14:33:57.207519 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:57.207566 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 14:33:57.889647 15665 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:57.889760 15665 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:33:57.889982 15665 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
W20260620 14:33:57.890388 15268 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
W20260620 14:33:57.890470 15270 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:33:57.890566 15270 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee; no voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09
I20260620 14:33:57.890643 15665 raft_consensus.cc:2726] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:58.051149 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:58.050926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:49785)
0620 14:33:58.051044 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:58.051085 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 14:33:58.088200 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:58.087994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:40673)
0620 14:33:58.088118 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:58.088153 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260620 14:33:58.208446 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:58.208187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:39175)
0620 14:33:58.208337 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:58.208383 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260620 14:33:59.051906 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:59.051672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:34573)
0620 14:33:59.051819 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:59.051857 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260620 14:33:59.088951 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:59.088659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:46773)
0620 14:33:59.088801 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:59.088875 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260620 14:33:59.209086 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:59.208896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:39793)
0620 14:33:59.209015 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:59.209045 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260620 14:34:00.052704 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:00.052479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:57447)
0620 14:34:00.052610 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:00.052644 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260620 14:34:00.089759 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:00.089548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:52343)
0620 14:34:00.089675 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:00.089714 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260620 14:34:00.209833 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:00.209613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:35705)
0620 14:34:00.209741 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:00.209785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 14:34:01.000713 15675 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:01.000808 15675 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:34:01.000972 15675 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
W20260620 14:34:01.001364 15270 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
W20260620 14:34:01.001495 15268 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:34:01.001531 15268 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee; no voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09
I20260620 14:34:01.001626 15675 raft_consensus.cc:2726] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:01.053342 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:01.053148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:59067)
0620 14:34:01.053263 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:01.053297 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260620 14:34:01.090389 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:01.090182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:52383)
0620 14:34:01.090307 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:01.090345 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260620 14:34:01.210506 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:01.210277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:48343)
0620 14:34:01.210420 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:01.210459 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260620 14:34:02.054049 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:02.053818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:42487)
0620 14:34:02.053958 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:02.054001 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260620 14:34:02.091027 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:02.090837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:56361)
0620 14:34:02.090947 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:02.090983 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260620 14:34:02.211243 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:02.211003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:55057)
0620 14:34:02.211149 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:02.211195 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 14:34:03.054809 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:03.054567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:41479)
0620 14:34:03.054716 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:03.054761 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 14:34:03.091639 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:03.091462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:43297)
0620 14:34:03.091569 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:03.091600 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260620 14:34:03.211969 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:03.211720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:52497)
0620 14:34:03.211880 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:03.211924 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 14:34:04.055490 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:04.055283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:50289)
0620 14:34:04.055410 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:04.055438 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260620 14:34:04.092310 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:04.092076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:52741)
0620 14:34:04.092216 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:04.092251 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260620 14:34:04.118893 15687 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:04.118980 15687 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:34:04.119122 15687 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
W20260620 14:34:04.119508 15270 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
W20260620 14:34:04.119589 15268 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:34:04.119669 15268 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee; no voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09
I20260620 14:34:04.119750 15687 raft_consensus.cc:2726] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:04.212671 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:04.212455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:39385)
0620 14:34:04.212592 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:04.212626 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260620 14:34:05.056195 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:05.055974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:44637)
0620 14:34:05.056109 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:05.056150 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260620 14:34:05.093022 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:05.092806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:39103)
0620 14:34:05.092942 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:05.092978 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260620 14:34:05.213377 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:05.213146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:52381)
0620 14:34:05.213292 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:05.213330 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260620 14:34:06.056886 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:06.056668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:49755)
0620 14:34:06.056804 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:06.056841 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 14:34:06.093673 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:06.093411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:52029)
0620 14:34:06.093597 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:06.093628 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":133,"threads_started":1}
I20260620 14:34:06.214146 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:06.213902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:41695)
0620 14:34:06.214051 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:06.214096 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260620 14:34:07.057624 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:07.057358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:37269)
0620 14:34:07.057531 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:07.057569 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260620 14:34:07.094502 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:07.094237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:56903)
0620 14:34:07.094408 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:07.094455 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260620 14:34:07.214848 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:07.214602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:58669)
0620 14:34:07.214761 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:07.214802 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260620 14:34:08.058427 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:08.058190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:52123)
0620 14:34:08.058330 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:08.058379 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260620 14:34:08.095211 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:08.095010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:60873)
0620 14:34:08.095129 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:08.095166 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 14:34:08.215644 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:08.215403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.3:38245)
0620 14:34:08.215525 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:08.215594 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260620 14:34:09.016508 15701 raft_consensus.cc:482] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:09.016604 15701 raft_consensus.cc:504] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2319e31784891a6a991a62f87ac09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43019 } } peers { permanent_uuid: "c245c8b1a20245cdad4618feedff7cee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40175 } } peers { permanent_uuid: "21451b18edca432291e22c3dac2ad168" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40731 } }
I20260620 14:34:09.016762 15701 leader_election.cc:288] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019), 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731)
W20260620 14:34:09.017175 15270 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3d2319e31784891a6a991a62f87ac09 (127.0.98.3:43019): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
W20260620 14:34:09.017251 15268 leader_election.cc:341] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21451b18edca432291e22c3dac2ad168 (127.0.98.2:40731): Not found: Tablet not found: e038aa58f9dd4dff8d981b3da6f9e75e
I20260620 14:34:09.017333 15268 leader_election.cc:302] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c245c8b1a20245cdad4618feedff7cee; no voters: 21451b18edca432291e22c3dac2ad168, f3d2319e31784891a6a991a62f87ac09
I20260620 14:34:09.017452 15701 raft_consensus.cc:2726] T e038aa58f9dd4dff8d981b3da6f9e75e P c245c8b1a20245cdad4618feedff7cee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:09.059283 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:09.059043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.1:44815)
0620 14:34:09.059201 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:09.059236 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260620 14:34:09.096004 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:09.095769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34007 (local address 127.0.98.2:53913)
0620 14:34:09.095919 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:09.095959 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260620 14:34:09.206972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15223
I20260620 14:34:09.211702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15386
I20260620 14:34:09.216215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15519
2026-06-20T14:34:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18253 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:09 dist-test-slave-tc2g krb5kdc[15706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:09 dist-test-slave-tc2g krb5kdc[15706](info): set up 2 sockets
Jun 20 14:34:09 dist-test-slave-tc2g krb5kdc[15706](info): commencing operation
krb5kdc: starting...
W20260620 14:34:10.257575 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:34:10.372046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:10.451472 15719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.451683 15719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.451706 15719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.452894 15719 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:34:10.452936 15719 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:34:10.452950 15719 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:34:10.452960 15719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:10.453001 15719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.453032 15719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.453049 15719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:10.453073 15719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:10.454567 15719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:10.454823 15719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.455052 15719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:10.460307 15724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.460351 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.460397 15725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.460373 15719 server_base.cc:1034] running on GCE node
I20260620 14:34:10.460806 15719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.461130 15719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.462268 15719 hybrid_clock.cc:648] HybridClock initialized: now 1781966050462259 us; error 33 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.464646 15719 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:34:10.465116 15719 webserver.cc:460] Webserver started at http://127.0.98.62:40107/ using document root <none> and password file <none>
I20260620 14:34:10.465346 15719 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.465399 15719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.465595 15719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:10.466704 15719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "b3654adde630406fb236e50950ff47e5"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "4cb698d6696ca52705107988fe293767"
server_key_iv: "4580232463b1f6a4a26bebbbe4370d78"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.467062 15719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3654adde630406fb236e50950ff47e5"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "4cb698d6696ca52705107988fe293767"
server_key_iv: "4580232463b1f6a4a26bebbbe4370d78"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.468377 15719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:34:10.469067 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.469240 15719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:34:10.469345 15719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "b3654adde630406fb236e50950ff47e5"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "4cb698d6696ca52705107988fe293767"
server_key_iv: "4580232463b1f6a4a26bebbbe4370d78"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.469453 15719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.495646 15719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.497650 15719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.497817 15719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.501966 15719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37501
I20260620 14:34:10.502000 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37501 every 8 connection(s)
I20260620 14:34:10.502346 15719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:10.503190 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:10.505316 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap starting.
I20260620 14:34:10.505956 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:10.506196 15787 log.cc:824] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:10.506731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15719
I20260620 14:34:10.506808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:34:10.506839 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: No bootstrap required, opened a new log
I20260620 14:34:10.507016 392 external_mini_cluster.cc:1428] Setting key 669cb2fc43468f0d2f3a53a2d4031d4d
I20260620 14:34:10.509014 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:10.509186 15787 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:10.509227 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3654adde630406fb236e50950ff47e5, State: Initialized, Role: FOLLOWER
I20260620 14:34:10.509366 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:10.509479 15787 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:10.509512 15787 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:10.509557 15787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:10.510215 15787 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:10.510315 15787 leader_election.cc:302] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3654adde630406fb236e50950ff47e5; no voters:
I20260620 14:34:10.510453 15787 leader_election.cc:288] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:10.510514 15792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:10.510675 15792 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 LEADER]: Becoming Leader. State: Replica: b3654adde630406fb236e50950ff47e5, State: Running, Role: LEADER
I20260620 14:34:10.510774 15792 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:10.510671 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:10.512084 15794 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3654adde630406fb236e50950ff47e5. Latest consensus state: current_term: 1 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:10.512095 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:10.512307 15794 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:10.512346 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:10.512643 15802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:10.513211 15802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:10.514981 15802 catalog_manager.cc:1347] Generated new cluster ID: 7f24aac627044fac95296e009942b30c
I20260620 14:34:10.515030 15802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:10.519215 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.507579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.0.98.62:37501)
0620 14:34:10.507763 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:10.507773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:10.507806 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:34:10.508237 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:10.508245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:10.508306 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:10.508465 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:10.509060 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.510757 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.511849 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.512488 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.512594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:10.515608 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:10.515612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:10.515612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:10.515623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:10.517548 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.517893 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.517895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.517896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.517986 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.518148 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.518149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.518149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.518418 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:10.518529 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:10.518904 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 14:34:10.519031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 14:34:10.538136 15802 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:10.538689 15802 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:10.544569 15802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Generated new TSK 0
I20260620 14:34:10.544759 15802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:34:10.575927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:10.656014 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.656219 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.656241 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.657408 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.657485 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.657512 15813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:34:10.659060 15813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:10.659313 15813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.659544 15813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:10.665117 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.665107 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.665153 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.665373 15813 server_base.cc:1034] running on GCE node
I20260620 14:34:10.665614 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.665809 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.666947 15813 hybrid_clock.cc:648] HybridClock initialized: now 1781966050666928 us; error 34 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.669363 15813 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:34:10.669803 15813 webserver.cc:460] Webserver started at http://127.0.98.1:35957/ using document root <none> and password file <none>
I20260620 14:34:10.670028 15813 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.670074 15813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.670243 15813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:10.671267 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "495a95a8a2474ca8951a6dc52998a48c"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "87d4241a1312513ea6f6602188e839cb"
server_key_iv: "7640eb110aecbb419fe61ed37b3527c6"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.671619 15813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "495a95a8a2474ca8951a6dc52998a48c"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "87d4241a1312513ea6f6602188e839cb"
server_key_iv: "7640eb110aecbb419fe61ed37b3527c6"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.673074 15813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:34:10.673902 15828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.674100 15813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:34:10.674199 15813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "495a95a8a2474ca8951a6dc52998a48c"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "87d4241a1312513ea6f6602188e839cb"
server_key_iv: "7640eb110aecbb419fe61ed37b3527c6"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.674284 15813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.687321 15813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.689109 15813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.689270 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.689587 15813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:10.689913 15813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:10.689967 15813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.690016 15813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:10.690042 15813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.695897 15813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260620 14:34:10.695945 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260620 14:34:10.696265 15813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:10.700762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15813
I20260620 14:34:10.700877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:10.701109 392 external_mini_cluster.cc:1428] Setting key adfe0e3039387b148cdc4a0ba2c213e1
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:10.705633 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.696949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44879 (local address 127.0.98.62:37501)
0620 14:34:10.696994 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:10.696996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:10.697196 (+ 200us) server_negotiation.cc:407] Connection header received
0620 14:34:10.697719 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:10.697720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:10.697730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:10.697786 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:10.698669 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.699165 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.700097 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.700283 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.700333 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:10.702734 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:10.702736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:10.702736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:10.702744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:10.704281 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.704646 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.704647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.704648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.704711 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.704932 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.704933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.704933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.705090 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:10.705165 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:10.705380 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:34:10.705501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:34:10.705705 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.696895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:44879)
0620 14:34:10.697060 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.697103 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:10.697638 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:10.697840 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:10.697843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:10.698406 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:10.698592 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:10.698597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.699242 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.699243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:10.700024 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:10.700025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.700358 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.700359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:10.700453 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:10.701034 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:10.701045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:10.702641 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:10.704355 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:10.704360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:10.704362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:10.704572 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:10.704777 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:10.704777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:10.704778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:10.704870 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:10.705158 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:10.705162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:10.705270 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:34:10.705347 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:34:10.705523 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260620 14:34:10.706207 15942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:10.706317 15942 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:10.706517 15942 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:10.707037 15751 ts_manager.cc:194] Registered new tserver with Master: 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:10.707634 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44879
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:34:10.726033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:10.806376 15947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.806577 15947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.806602 15947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.807739 15947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.807780 15947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.807830 15947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:34:10.809221 15947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:10.809491 15947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.809742 15947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:10.814671 15947 server_base.cc:1034] running on GCE node
W20260620 14:34:10.814656 15953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.814651 15955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.814646 15952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.815080 15947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.815326 15947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.816488 15947 hybrid_clock.cc:648] HybridClock initialized: now 1781966050816485 us; error 38 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.818898 15947 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:34:10.819372 15947 webserver.cc:460] Webserver started at http://127.0.98.2:46851/ using document root <none> and password file <none>
I20260620 14:34:10.819594 15947 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.819661 15947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.819808 15947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:10.820883 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "022f93ed34734317aa3c3f57afc75c92"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "99d2d1457c386f0e14a0f54385a74215"
server_key_iv: "8c862579eebf8e3e0d0f213f8e47abca"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.821171 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "022f93ed34734317aa3c3f57afc75c92"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "99d2d1457c386f0e14a0f54385a74215"
server_key_iv: "8c862579eebf8e3e0d0f213f8e47abca"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.822479 15947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:34:10.823125 15962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.823302 15947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:34:10.823386 15947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "022f93ed34734317aa3c3f57afc75c92"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "99d2d1457c386f0e14a0f54385a74215"
server_key_iv: "8c862579eebf8e3e0d0f213f8e47abca"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.823449 15947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.843783 15947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.845672 15947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.845788 15947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.845983 15947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:10.846222 15947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:10.846251 15947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.846273 15947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:10.846285 15947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.851756 15947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39535
I20260620 14:34:10.851837 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39535 every 8 connection(s)
I20260620 14:34:10.852159 15947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:10.860603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 15947
I20260620 14:34:10.860702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:10.860901 392 external_mini_cluster.cc:1428] Setting key b3f8fb6f561245243e8adf69af8d683f
I20260620 14:34:10.861263 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.852926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34265 (local address 127.0.98.62:37501)
0620 14:34:10.853021 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:10.853024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:10.853174 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:34:10.853755 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:10.853756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:10.853766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:10.853819 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:10.854714 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.855213 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.856182 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.856367 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.856417 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:10.858449 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:10.858450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:10.858450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:10.858457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:10.859790 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.860184 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.860185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.860186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.860243 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:10.860502 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:10.860504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:10.860504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:10.860651 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:10.860731 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:10.861084 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 14:34:10.861175 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:34:10.861528 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.852873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:34265)
0620 14:34:10.853056 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.853093 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:10.853662 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:10.853873 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:10.853877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:10.854447 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:10.854636 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:10.854641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.855287 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.855288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:10.856107 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:10.856109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:10.856454 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:10.856455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:10.856549 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:10.856989 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:10.856997 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:10.858366 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:10.859878 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:10.859883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:10.859884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:10.860090 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:10.860311 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:10.860311 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:10.860311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:10.860404 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:10.860730 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:10.860736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:10.860875 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:34:10.861057 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 14:34:10.861226 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260620 14:34:10.862027 16076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:10.862123 16076 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:10.862310 16076 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:10.862681 15751 ts_manager.cc:194] Registered new tserver with Master: 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
I20260620 14:34:10.863180 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34265
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:34:10.884279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:10.963218 16081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.963477 16081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.963515 16081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.964689 16081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.964756 16081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.964802 16081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:34:10.966272 16081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:10.966531 16081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.966780 16081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:10.971626 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.971616 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.971642 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.971668 16081 server_base.cc:1034] running on GCE node
I20260620 14:34:10.971978 16081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.972158 16081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.973291 16081 hybrid_clock.cc:648] HybridClock initialized: now 1781966050973261 us; error 51 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.975674 16081 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:34:10.976109 16081 webserver.cc:460] Webserver started at http://127.0.98.3:44711/ using document root <none> and password file <none>
I20260620 14:34:10.976341 16081 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.976383 16081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.976563 16081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:10.977741 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "33a2585d480b46caa0702b6d18bfb507"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "a1f7c848cc6ef526cceb8fa03a25fdfa"
server_key_iv: "58d2a01fbb55580fd5a0de3a25288b04"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.978087 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33a2585d480b46caa0702b6d18bfb507"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "a1f7c848cc6ef526cceb8fa03a25fdfa"
server_key_iv: "58d2a01fbb55580fd5a0de3a25288b04"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.979367 16081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:34:10.980047 16096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.980213 16081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:34:10.980309 16081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "33a2585d480b46caa0702b6d18bfb507"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "a1f7c848cc6ef526cceb8fa03a25fdfa"
server_key_iv: "58d2a01fbb55580fd5a0de3a25288b04"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.980393 16081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:11.010915 16081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:11.013547 16081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:11.013715 16081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:11.013976 16081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:11.014355 16081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:11.014400 16081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:11.014448 16081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:11.014477 16081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:11.020597 16081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33711
I20260620 14:34:11.020629 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33711 every 8 connection(s)
I20260620 14:34:11.020999 16081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:34:11 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:11.029143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16081
I20260620 14:34:11.029251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:11.029521 392 external_mini_cluster.cc:1428] Setting key 8bdde262e644df0ce6c1a58a100fd7d0
I20260620 14:34:11.030649 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.021753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55031 (local address 127.0.98.62:37501)
0620 14:34:11.021829 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.021831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.022057 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:34:11.022567 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.022568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.022577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.022629 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:11.023598 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.024069 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.025175 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.025354 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.025398 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.027680 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:11.027681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:11.027681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:11.027688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:11.029080 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:11.029593 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:11.029594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:11.029594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:11.029651 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:11.029990 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:11.029991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:11.029991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:11.030139 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:11.030199 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.030522 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.030589 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:34:11.030894 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.021739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:55031)
0620 14:34:11.021933 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.021971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.022489 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.022688 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.022691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.023323 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:11.023525 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.023530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.024157 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.024158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.025095 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.025097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.025480 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.025482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.025616 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.026062 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:11.026071 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:11.027586 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:11.029161 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:11.029167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:11.029170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:11.029517 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:11.029726 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:11.029726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:11.029727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:11.029923 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:11.030209 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:11.030215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:11.030378 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.030497 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.030653 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260620 14:34:11.031370 16210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:11.031472 16210 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:11.031644 16210 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:11.031975 15751 ts_manager.cc:194] Registered new tserver with Master: 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711)
I20260620 14:34:11.032342 15751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55031
I20260620 14:34:11.041172 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:11.048048 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.042578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.0.98.62:37501)
0620 14:34:11.042627 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.042629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.042730 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:34:11.042769 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.042770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.042777 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.042828 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:11.043330 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.043785 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.044466 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.044742 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.044807 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.045930 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:11.045931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:11.045931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:11.045936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:11.047227 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:11.047501 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:11.047502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:11.047502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:11.047547 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:11.047657 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:11.047657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:11.047658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:11.047774 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:11.047842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.047935 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.047989 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:34:11.049530 15751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:11.051070 15751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:11.057598 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:52446)
0620 14:34:11.053929 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.053937 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.053981 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.054264 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.054265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.054267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:11.054309 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.054311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.055185 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.055185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.056386 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.056387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.057318 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.057319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.057379 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.057382 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.057453 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.057509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260620 14:34:11.057933 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.98.1:45115)
0620 14:34:11.053702 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.053707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.054040 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:34:11.054085 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.054088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.054104 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.054183 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:11.054375 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.055035 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.056460 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.057173 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.057233 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.057288 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.057782 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.057832 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260620 14:34:11.057935 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.0.98.3:33711)
0620 14:34:11.053494 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.053499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.053788 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:34:11.053893 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.053895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.053930 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.054000 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:11.054292 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.055006 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.056301 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.057045 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.057118 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.057176 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.057440 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.057535 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260620 14:34:11.058670 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.0.1:51348)
0620 14:34:11.053688 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.053730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.053807 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.054088 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.054092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.054096 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:11.054202 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.054205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.056359 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.056359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.057130 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.057133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.058390 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.058391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.058478 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.058483 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.058537 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.058578 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":23,"thread_start_us":80,"threads_started":1}
I20260620 14:34:11.058734 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.0.1:60928)
0620 14:34:11.053688 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.053730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.053807 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.054088 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.054092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.054095 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:11.054202 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.054204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.055122 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.055123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.056220 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.056222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.057214 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.057215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.057295 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.057310 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.057382 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.058548 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":54,"thread_start_us":64,"threads_started":1}
I20260620 14:34:11.058773 15876 tablet_service.cc:1467] Processing CreateTablet for tablet fbf29d66d93f492a897958ffa876cc51 (DEFAULT_TABLE table=test-table [id=ce5c2338dbe348b2966c1b0253de29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:11.058974 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.053527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.98.2:39535)
0620 14:34:11.053702 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.053707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.053785 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:34:11.053893 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.053895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.053911 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.053999 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:11.055103 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.056248 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.057212 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.057979 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.058295 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.058364 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.058841 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.058887 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260620 14:34:11.059155 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf29d66d93f492a897958ffa876cc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:11.059358 16144 tablet_service.cc:1467] Processing CreateTablet for tablet fbf29d66d93f492a897958ffa876cc51 (DEFAULT_TABLE table=test-table [id=ce5c2338dbe348b2966c1b0253de29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:11.059571 16010 tablet_service.cc:1467] Processing CreateTablet for tablet fbf29d66d93f492a897958ffa876cc51 (DEFAULT_TABLE table=test-table [id=ce5c2338dbe348b2966c1b0253de29ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:11.059671 16144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf29d66d93f492a897958ffa876cc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:11.059890 16010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf29d66d93f492a897958ffa876cc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:11.062460 16227 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Bootstrap starting.
I20260620 14:34:11.062831 16229 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Bootstrap starting.
I20260620 14:34:11.062933 16228 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap starting.
I20260620 14:34:11.063316 16227 tablet_bootstrap.cc:654] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:11.063460 16229 tablet_bootstrap.cc:654] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:11.063568 16228 tablet_bootstrap.cc:654] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:11.063961 16227 log.cc:824] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:11.063972 16229 log.cc:824] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:11.064133 16228 log.cc:824] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:11.064879 16227 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: No bootstrap required, opened a new log
I20260620 14:34:11.064879 16228 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: No bootstrap required, opened a new log
I20260620 14:34:11.064954 16227 ts_tablet_manager.cc:1397] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:34:11.064954 16228 ts_tablet_manager.cc:1397] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:11.065738 16229 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: No bootstrap required, opened a new log
I20260620 14:34:11.065822 16229 ts_tablet_manager.cc:1397] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:34:11.066957 16228 raft_consensus.cc:348] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.066957 16227 raft_consensus.cc:348] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.067148 16227 raft_consensus.cc:374] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:11.067148 16228 raft_consensus.cc:374] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:11.067188 16228 raft_consensus.cc:729] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495a95a8a2474ca8951a6dc52998a48c, State: Initialized, Role: FOLLOWER
I20260620 14:34:11.067188 16227 raft_consensus.cc:729] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022f93ed34734317aa3c3f57afc75c92, State: Initialized, Role: FOLLOWER
I20260620 14:34:11.067107 16229 raft_consensus.cc:348] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.067312 16228 consensus_queue.cc:260] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.067312 16227 consensus_queue.cc:260] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.067346 16229 raft_consensus.cc:374] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:11.067415 16229 raft_consensus.cc:729] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a2585d480b46caa0702b6d18bfb507, State: Initialized, Role: FOLLOWER
I20260620 14:34:11.067552 16229 consensus_queue.cc:260] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.067590 16228 ts_tablet_manager.cc:1428] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:11.067590 16227 ts_tablet_manager.cc:1428] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:34:11.067708 16076 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
I20260620 14:34:11.067829 15942 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
I20260620 14:34:11.067826 16229 ts_tablet_manager.cc:1428] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:11.068068 16210 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
W20260620 14:34:11.102757 16077 tablet.cc:2369] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:11.129801 16233 raft_consensus.cc:482] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:11.129944 16233 raft_consensus.cc:504] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.130311 16233 leader_election.cc:288] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:11.133163 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.130609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39525 (local address 127.0.98.1:45115)
0620 14:34:11.130650 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.130653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.130661 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:34:11.130719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.130720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.130732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.130792 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:11.130968 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.131472 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132435 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.132804 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132857 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.132866 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.133049 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.133089 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:34:11.133200 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.130576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49543 (local address 127.0.98.3:33711)
0620 14:34:11.130623 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.130625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.130785 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:34:11.130816 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.130817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.130830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.130887 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:11.131064 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.131482 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132466 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.132821 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132910 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.132919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.133094 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.133138 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:34:11.133190 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.130506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.98.2:39525)
0620 14:34:11.130594 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.130606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.130654 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.130812 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.130813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.130815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:11.130888 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.130890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.131562 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.131563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.132333 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.132335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.132888 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.132971 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.132979 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.133024 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.133083 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:34:11.133208 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.130524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.98.2:49543)
0620 14:34:11.130690 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.130700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:11.130760 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:11.130895 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:11.130895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:11.130897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:11.130992 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.130993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.131594 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.131595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.132347 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:11.132348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.132942 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.132943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:11.132993 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.132997 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:34:11.133070 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:34:11.133105 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
I20260620 14:34:11.133658 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf29d66d93f492a897958ffa876cc51" candidate_uuid: "022f93ed34734317aa3c3f57afc75c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495a95a8a2474ca8951a6dc52998a48c" is_pre_election: true
I20260620 14:34:11.133823 15896 raft_consensus.cc:2454] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022f93ed34734317aa3c3f57afc75c92 in term 0.
I20260620 14:34:11.133795 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf29d66d93f492a897958ffa876cc51" candidate_uuid: "022f93ed34734317aa3c3f57afc75c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a2585d480b46caa0702b6d18bfb507" is_pre_election: true
I20260620 14:34:11.133976 16164 raft_consensus.cc:2454] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022f93ed34734317aa3c3f57afc75c92 in term 0.
I20260620 14:34:11.134217 15965 leader_election.cc:302] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507; no voters:
I20260620 14:34:11.134363 16233 raft_consensus.cc:2781] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:11.134418 16233 raft_consensus.cc:482] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:11.134469 16233 raft_consensus.cc:3037] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:11.135151 16233 raft_consensus.cc:504] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.135288 16233 leader_election.cc:288] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [CANDIDATE]: Term 1 election: Requested vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:11.135423 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf29d66d93f492a897958ffa876cc51" candidate_uuid: "022f93ed34734317aa3c3f57afc75c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495a95a8a2474ca8951a6dc52998a48c"
I20260620 14:34:11.135488 15896 raft_consensus.cc:3037] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:11.136215 15896 raft_consensus.cc:2454] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022f93ed34734317aa3c3f57afc75c92 in term 1.
I20260620 14:34:11.136336 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf29d66d93f492a897958ffa876cc51" candidate_uuid: "022f93ed34734317aa3c3f57afc75c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a2585d480b46caa0702b6d18bfb507"
I20260620 14:34:11.136385 15966 leader_election.cc:302] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f93ed34734317aa3c3f57afc75c92, 495a95a8a2474ca8951a6dc52998a48c; no voters:
I20260620 14:34:11.136426 16164 raft_consensus.cc:3037] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:11.136497 16233 raft_consensus.cc:2781] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:11.136684 16233 raft_consensus.cc:686] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 1 LEADER]: Becoming Leader. State: Replica: 022f93ed34734317aa3c3f57afc75c92, State: Running, Role: LEADER
I20260620 14:34:11.136778 16233 consensus_queue.cc:237] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:11.137303 16164 raft_consensus.cc:2454] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022f93ed34734317aa3c3f57afc75c92 in term 1.
I20260620 14:34:11.137674 15751 catalog_manager.cc:5495] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "022f93ed34734317aa3c3f57afc75c92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } health_report { overall_health: HEALTHY } } }
I20260620 14:34:11.148485 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.146470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.98.2:39535)
0620 14:34:11.146542 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:11.146544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:11.146551 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:34:11.146581 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:11.146581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:11.146603 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:11.146665 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:11.146777 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.147211 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.147887 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:11.148060 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:11.148107 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:11.148229 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:11.148315 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:11.148371 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:11.148402 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 14:34:11.148423 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:34:11.150077 15896 raft_consensus.cc:1261] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Refusing update from remote peer 022f93ed34734317aa3c3f57afc75c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:11.150130 16164 raft_consensus.cc:1261] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Refusing update from remote peer 022f93ed34734317aa3c3f57afc75c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:11.150357 16233 consensus_queue.cc:1035] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:11.150532 16237 consensus_queue.cc:1035] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:11.154942 15750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:11.155040 15750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:11.155735 15750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Sending DeleteTablet for 3 replicas of tablet fbf29d66d93f492a897958ffa876cc51
I20260620 14:34:11.156044 16144 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf29d66d93f492a897958ffa876cc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60928
I20260620 14:34:11.156064 15876 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf29d66d93f492a897958ffa876cc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52446
I20260620 14:34:11.156141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15813
I20260620 14:34:11.156243 16010 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf29d66d93f492a897958ffa876cc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51348
I20260620 14:34:11.158236 16251 tablet_replica.cc:330] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: stopping tablet replica
I20260620 14:34:11.158339 16251 raft_consensus.cc:2229] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:11.158427 16251 raft_consensus.cc:2258] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:11.158587 16250 tablet_replica.cc:330] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: stopping tablet replica
I20260620 14:34:11.158679 16250 raft_consensus.cc:2229] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:11.158771 16250 raft_consensus.cc:2258] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:11.158856 16251 ts_tablet_manager.cc:1905] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:11.160290 16251 ts_tablet_manager.cc:1918] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:11.160378 16251 log.cc:1196] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/fbf29d66d93f492a897958ffa876cc51
I20260620 14:34:11.160667 16251 ts_tablet_manager.cc:1939] T fbf29d66d93f492a897958ffa876cc51 P 022f93ed34734317aa3c3f57afc75c92: Deleting consensus metadata
I20260620 14:34:11.161075 15736 catalog_manager.cc:4841] TS 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): tablet fbf29d66d93f492a897958ffa876cc51 (table test-table [id=ce5c2338dbe348b2966c1b0253de29ca]) successfully deleted
I20260620 14:34:11.161772 16250 ts_tablet_manager.cc:1905] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:34:11.162920 15738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45115 (error 108)
I20260620 14:34:11.162936 16250 ts_tablet_manager.cc:1918] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:11.162995 16250 log.cc:1196] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/fbf29d66d93f492a897958ffa876cc51
I20260620 14:34:11.163264 16250 ts_tablet_manager.cc:1939] T fbf29d66d93f492a897958ffa876cc51 P 33a2585d480b46caa0702b6d18bfb507: Deleting consensus metadata
I20260620 14:34:11.163602 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.163501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:52450)
0620 14:34:11.163542 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.163574 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:34:11.163610 15737 catalog_manager.cc:4841] TS 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): tablet fbf29d66d93f492a897958ffa876cc51 (table test-table [id=ce5c2338dbe348b2966c1b0253de29ca]) successfully deleted
W20260620 14:34:11.163722 15738 catalog_manager.cc:4584] TS 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbf29d66d93f492a897958ffa876cc51: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
I20260620 14:34:11.163890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15947
I20260620 14:34:11.168632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16081
I20260620 14:34:11.173074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 15719
I20260620 14:34:31.178491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--webserver_interface=127.0.98.62
--webserver_port=40107
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:31.258111 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.258306 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.258327 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.259447 16253 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:34:31.259485 16253 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:34:31.259498 16253 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:34:31.259510 16253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:31.259522 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.259562 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.259591 16253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:31.259608 16253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:31.261042 16253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:31.261291 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.261539 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:31.270193 16253 server_base.cc:1034] running on GCE node
W20260620 14:34:31.270190 16259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.270136 16260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.270148 16262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.270582 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.270938 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.272105 16253 hybrid_clock.cc:648] HybridClock initialized: now 1781966071272084 us; error 32 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.280390 16253 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:34:31.280872 16253 webserver.cc:460] Webserver started at http://127.0.98.62:40107/ using document root <none> and password file <none>
I20260620 14:34:31.281136 16253 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.281221 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.282673 16253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:31.283366 16269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.283536 16253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:34:31.283638 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "b3654adde630406fb236e50950ff47e5"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "4cb698d6696ca52705107988fe293767"
server_key_iv: "4580232463b1f6a4a26bebbbe4370d78"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.284058 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.297919 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.299746 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.299916 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.304025 16253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37501
I20260620 14:34:31.304055 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37501 every 8 connection(s)
I20260620 14:34:31.304441 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:31.305963 16322 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:34:31.306553 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap starting.
I20260620 14:34:31.313601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16253
I20260620 14:34:31.313905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45115
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=35957
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:31.314867 16322 log.cc:824] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.316752 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:31.317102 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap complete.
I20260620 14:34:31.319403 16322 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:31.319653 16322 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3654adde630406fb236e50950ff47e5, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.319856 16322 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:31.319952 16322 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:31.320019 16322 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:31.320066 16322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:34:31.321055 16322 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:31.321177 16322 leader_election.cc:302] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3654adde630406fb236e50950ff47e5; no voters:
I20260620 14:34:31.321379 16322 leader_election.cc:288] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:34:31.321444 16326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:34:31.321594 16326 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 LEADER]: Becoming Leader. State: Replica: b3654adde630406fb236e50950ff47e5, State: Running, Role: LEADER
I20260620 14:34:31.321684 16322 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:31.321681 16326 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:31.322216 16328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3654adde630406fb236e50950ff47e5. Latest consensus state: current_term: 2 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:31.322465 16328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:31.322777 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:31.323042 16338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:31.323068 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:31.323848 16338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:31.323951 16338 catalog_manager.cc:1259] Loaded cluster ID: 7f24aac627044fac95296e009942b30c
I20260620 14:34:31.323985 16338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:31.325098 16338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:31.325273 16338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Loaded TSK: 0
I20260620 14:34:31.338246 16338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Generated new TSK 1
I20260620 14:34:31.338363 16338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:34:31.396474 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.396704 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.396745 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.397957 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.398021 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.398067 16324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:34:31.399534 16324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35957
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:31.399794 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.400027 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.404971 16349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.404971 16350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.405087 16324 server_base.cc:1034] running on GCE node
W20260620 14:34:31.405208 16352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.405611 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.405893 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.407073 16324 hybrid_clock.cc:648] HybridClock initialized: now 1781966071407058 us; error 28 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.410218 16324 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:34:31.410733 16324 webserver.cc:460] Webserver started at http://127.0.98.1:35957/ using document root <none> and password file <none>
I20260620 14:34:31.410998 16324 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.411078 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.412482 16324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:34:31.413240 16359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.413411 16324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:34:31.413516 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "495a95a8a2474ca8951a6dc52998a48c"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "87d4241a1312513ea6f6602188e839cb"
server_key_iv: "7640eb110aecbb419fe61ed37b3527c6"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.413859 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.425014 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.427274 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.427435 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.427676 16324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.428124 16366 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:34:31.429996 16324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:34:31.430059 16324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:34:31.430116 16324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:34:31.430716 16324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:34:31.430773 16324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:34:31.430840 16366 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap starting.
I20260620 14:34:31.438055 16324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260620 14:34:31.438177 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260620 14:34:31.438475 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:31.438817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16324
I20260620 14:34:31.439085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39535
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=46851
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:31.443279 16366 log.cc:824] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:31.451925 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.441067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:54635)
0620 14:34:31.441767 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.441817 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.442335 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.442827 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.442832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.443124 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.443308 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.443313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.444686 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.444687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.445619 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.445622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.446341 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.446342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.446435 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.446977 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.446985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.448369 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.450306 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.450313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.450315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.450546 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.450803 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.450803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.450804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.450913 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.451280 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.451287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.451426 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.451532 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.451710 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":91,"threads_started":1}
I20260620 14:34:31.451938 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.441142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54635 (local address 127.0.98.62:37501)
0620 14:34:31.441563 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.441571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.441926 (+ 355us) server_negotiation.cc:407] Connection header received
0620 14:34:31.442565 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.442570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.442614 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.442749 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.443468 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.444495 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.445722 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.446157 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.446224 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.448502 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.448506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.448506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.448515 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.450202 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.450630 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.450634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.450634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.450729 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.450986 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.450987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.450987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.451205 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.451300 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.451583 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.451710 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":90,"threads_started":1}
I20260620 14:34:31.452970 16474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:31.453090 16474 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.453267 16474 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:31.454071 16286 ts_manager.cc:194] Registered new tserver with Master: 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:31.455160 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54635
I20260620 14:34:31.455859 16366 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:31.456238 16366 tablet_bootstrap.cc:492] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap complete.
I20260620 14:34:31.457460 16366 ts_tablet_manager.cc:1397] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20260620 14:34:31.458865 16366 raft_consensus.cc:348] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:31.459183 16366 raft_consensus.cc:729] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495a95a8a2474ca8951a6dc52998a48c, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.459344 16366 consensus_queue.cc:260] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } }
I20260620 14:34:31.459623 16366 ts_tablet_manager.cc:1428] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:31.459637 16474 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
I20260620 14:34:31.460965 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.454635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:56742)
0620 14:34:31.457559 (+ 2924us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.457599 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.457674 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.457856 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.457858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.457860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.457919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.457921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.458728 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.458729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.459809 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.459812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.460708 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.460708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.460761 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.460769 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.460846 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.460896 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2874,"thread_start_us":92,"threads_started":1}
I20260620 14:34:31.461004 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.454877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.98.1:45115)
0620 14:34:31.455038 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.455042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.457656 (+ 2614us) server_negotiation.cc:407] Connection header received
0620 14:34:31.457751 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.457754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.457769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.457852 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.457991 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.458643 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.459901 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.460569 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.460627 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.460661 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.460904 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.460942 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260620 14:34:31.461537 16392 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf29d66d93f492a897958ffa876cc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56742
I20260620 14:34:31.461967 16483 tablet_replica.cc:330] stopping tablet replica
I20260620 14:34:31.462076 16483 raft_consensus.cc:2229] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:31.462137 16483 raft_consensus.cc:2258] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.462448 16483 ts_tablet_manager.cc:1905] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:31.464464 16483 ts_tablet_manager.cc:1918] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:31.464558 16483 log.cc:1196] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal/wals/fbf29d66d93f492a897958ffa876cc51
I20260620 14:34:31.464776 16483 ts_tablet_manager.cc:1939] T fbf29d66d93f492a897958ffa876cc51 P 495a95a8a2474ca8951a6dc52998a48c: Deleting consensus metadata
I20260620 14:34:31.465246 16273 catalog_manager.cc:4841] TS 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115): tablet fbf29d66d93f492a897958ffa876cc51 (table test-table [id=ce5c2338dbe348b2966c1b0253de29ca]) successfully deleted
W20260620 14:34:31.537297 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.537533 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.537559 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.538651 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.538684 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.538709 16476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:34:31.540113 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46851
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:31.540313 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.540482 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.545591 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.545655 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.545595 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.545827 16476 server_base.cc:1034] running on GCE node
I20260620 14:34:31.546056 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.546314 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.547458 16476 hybrid_clock.cc:648] HybridClock initialized: now 1781966071547437 us; error 44 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.550005 16476 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:34:31.550525 16476 webserver.cc:460] Webserver started at http://127.0.98.2:46851/ using document root <none> and password file <none>
I20260620 14:34:31.550797 16476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.550881 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.552156 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:31.552726 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.552935 16476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:34:31.553033 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "022f93ed34734317aa3c3f57afc75c92"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "99d2d1457c386f0e14a0f54385a74215"
server_key_iv: "8c862579eebf8e3e0d0f213f8e47abca"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.553369 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.566655 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.568476 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.568632 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.568861 16476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.569164 16476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:31.569221 16476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.569264 16476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:31.569293 16476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.575335 16476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39535
I20260620 14:34:31.575389 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39535 every 8 connection(s)
I20260620 14:34:31.575773 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:31.584956 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.576489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44525 (local address 127.0.98.62:37501)
0620 14:34:31.576569 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.576571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.576777 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:34:31.577303 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.577304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.577318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.577390 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.578373 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.578885 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.579901 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.580133 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.580217 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.582328 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.582329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.582330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.582338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.583555 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.584000 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.584001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.584002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.584063 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.584313 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.584314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.584314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.584430 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.584505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.584765 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.584867 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:34:31.585130 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.576487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:44525)
0620 14:34:31.576664 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.576698 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.577196 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.577510 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.577514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.578079 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.578262 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.578267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.578986 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.578987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.579774 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.579775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.580228 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.580229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.580344 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.580795 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.580803 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.582214 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.583643 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.583649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.583652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.583907 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.584133 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.584134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.584134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.584237 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.584497 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.584503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.584638 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.584746 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.584909 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260620 14:34:31.585732 16612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:31.585837 16612 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.585872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16476
I20260620 14:34:31.586036 16612 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:31.586158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33711
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=44711
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:31.586457 16286 ts_manager.cc:194] Registered new tserver with Master: 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
I20260620 14:34:31.586901 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44525
W20260620 14:34:31.666891 16615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.667083 16615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.667106 16615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.668259 16615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.668309 16615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.668336 16615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:34:31.669884 16615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44711
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:31.670128 16615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.670356 16615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.675376 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.675357 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.675343 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.675390 16615 server_base.cc:1034] running on GCE node
I20260620 14:34:31.675742 16615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.675935 16615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.677135 16615 hybrid_clock.cc:648] HybridClock initialized: now 1781966071677118 us; error 47 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.679721 16615 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:34:31.680190 16615 webserver.cc:460] Webserver started at http://127.0.98.3:44711/ using document root <none> and password file <none>
I20260620 14:34:31.680446 16615 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.680532 16615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.681921 16615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:31.682484 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.682655 16615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:34:31.682751 16615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "33a2585d480b46caa0702b6d18bfb507"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "a1f7c848cc6ef526cceb8fa03a25fdfa"
server_key_iv: "58d2a01fbb55580fd5a0de3a25288b04"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.683094 16615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.693230 16615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.695559 16615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.695744 16615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.695966 16615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.696259 16615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:31.696316 16615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.696367 16615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:31.696389 16615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.702219 16615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33711
I20260620 14:34:31.702265 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33711 every 8 connection(s)
I20260620 14:34:31.702598 16615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:34:31 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:34:31.711128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16615
I20260620 14:34:31.711524 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.703323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60311 (local address 127.0.98.62:37501)
0620 14:34:31.703411 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.703413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.703594 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:34:31.704048 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.704048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.704059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.704106 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.704997 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.705556 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.706525 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.706720 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.706790 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.708780 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.708781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.708782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.708789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.710088 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.710526 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.710527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.710527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.710585 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.710845 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.710846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.710846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.710956 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.711026 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.711350 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.711431 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:34:31.711725 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.703285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:60311)
0620 14:34:31.703472 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.703506 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.703965 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.704166 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.704169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.704722 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.704902 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.704907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.705633 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.705634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.706446 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.706448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.706808 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.706809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.706921 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.707362 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.707370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.708684 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.710198 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.710205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.710207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.710431 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.710654 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.710655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.710655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.710759 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.711029 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.711036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.711180 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.711290 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.711489 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260620 14:34:31.712231 16744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37501
I20260620 14:34:31.712344 16744 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.712543 16744 heartbeater.cc:511] Master 127.0.98.62:37501 requested a full tablet report, sending...
I20260620 14:34:31.712940 16286 ts_manager.cc:194] Registered new tserver with Master: 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711)
I20260620 14:34:31.713305 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60311
I20260620 14:34:31.717990 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.711556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.0.98.62:37501)
0620 14:34:31.711769 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.711771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.711779 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:34:31.711861 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.711861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.711872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.711932 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.712802 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.713305 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.714367 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.714539 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.714610 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.715752 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.715753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.715753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.715761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.716902 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.717248 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.717249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.717250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.717305 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.717518 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.717518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.717519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.717617 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.717789 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.717823 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.717893 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260620 14:34:31.718669 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:31.721347 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.719019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.98.62:37501)
0620 14:34:31.719059 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.719061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.719169 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:34:31.719211 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.719212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.719235 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.719290 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:31.719391 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.719801 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.720626 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.720792 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.720872 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.721019 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:31.721108 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:31.721161 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.721251 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.721269 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:34:31.721621 16286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:31.723416 16286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:31.725555 16392 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf (DEFAULT_TABLE table=test-table [id=f230b7893af54123a5141c1b2d11d3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.725751 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3964bcbe874eb781f9a5718c29d2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.727174 16366 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap starting.
I20260620 14:34:31.727494 16366 tablet_bootstrap.cc:654] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.728031 16366 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: No bootstrap required, opened a new log
I20260620 14:34:31.728084 16366 ts_tablet_manager.cc:1397] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:34:31.728181 16366 raft_consensus.cc:348] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.728250 16366 raft_consensus.cc:374] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.728294 16366 raft_consensus.cc:729] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495a95a8a2474ca8951a6dc52998a48c, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.728343 16366 consensus_queue.cc:260] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.728431 16366 ts_tablet_manager.cc:1428] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.729152 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.0.1:53814)
0620 14:34:31.725354 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.725364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.725402 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.725853 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.725854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.725856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.725895 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.725896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.726845 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.726846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.727926 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.727929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.728897 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.728898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.728963 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.728965 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.729044 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.729084 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:34:31.729283 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.725480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.0.98.3:33711)
0620 14:34:31.725673 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.725678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.725690 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:34:31.725728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.725731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.725750 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.725868 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.725961 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.726768 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.728011 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.728726 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.728793 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.728864 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.729153 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.729201 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260620 14:34:31.729965 16678 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf (DEFAULT_TABLE table=test-table [id=f230b7893af54123a5141c1b2d11d3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.730221 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3964bcbe874eb781f9a5718c29d2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.731024 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.0.1:53626)
0620 14:34:31.725722 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.725731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.725780 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.727513 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.727514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.727517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.727568 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.727570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.729017 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.729019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.729985 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.729987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.730765 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.730766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.730831 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.730844 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.730908 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.730946 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":20,"thread_start_us":123,"threads_started":1}
I20260620 14:34:31.731096 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.725512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.0.98.2:39535)
0620 14:34:31.727347 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.727352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.727362 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:34:31.727398 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.727400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.727414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.727495 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.727633 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.728849 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.730079 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.730677 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.730736 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.730769 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.730945 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.730997 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":116,"threads_started":1}
I20260620 14:34:31.732964 16754 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Bootstrap starting.
I20260620 14:34:31.733691 16754 tablet_bootstrap.cc:654] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.733683 16546 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf (DEFAULT_TABLE table=test-table [id=f230b7893af54123a5141c1b2d11d3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.734052 16546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3964bcbe874eb781f9a5718c29d2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.734277 16754 log.cc:824] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.735018 16754 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: No bootstrap required, opened a new log
I20260620 14:34:31.735085 16754 ts_tablet_manager.cc:1397] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:31.737049 16754 raft_consensus.cc:348] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.737233 16754 raft_consensus.cc:374] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.737270 16754 raft_consensus.cc:729] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a2585d480b46caa0702b6d18bfb507, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.737442 16754 consensus_queue.cc:260] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.737627 16756 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Bootstrap starting.
I20260620 14:34:31.737708 16754 ts_tablet_manager.cc:1428] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:31.737849 16744 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
I20260620 14:34:31.738308 16756 tablet_bootstrap.cc:654] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.738880 16756 log.cc:824] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.739532 16756 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: No bootstrap required, opened a new log
I20260620 14:34:31.739600 16756 ts_tablet_manager.cc:1397] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:31.740957 16756 raft_consensus.cc:348] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.741109 16756 raft_consensus.cc:374] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.741132 16756 raft_consensus.cc:729] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022f93ed34734317aa3c3f57afc75c92, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.741237 16756 consensus_queue.cc:260] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.741528 16756 ts_tablet_manager.cc:1428] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:31.741580 16612 heartbeater.cc:503] Master 127.0.98.62:37501 was elected leader, sending a full tablet report...
W20260620 14:34:31.826318 16613 tablet.cc:2369] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:31.835029 16757 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:31.835170 16757 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.835582 16757 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535), 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:31.838807 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.835808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33565 (local address 127.0.98.2:39535)
0620 14:34:31.835864 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.835867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.835921 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:34:31.835952 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.835953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.835966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.836044 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.836194 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.836927 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.837981 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.838449 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.838504 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.838514 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.838698 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.838742 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:34:31.838802 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.835742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.98.3:33565)
0620 14:34:31.835798 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.835811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.835857 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.836058 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.836058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.836060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.836141 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.836143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.837007 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.837008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.837893 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.837895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.838531 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.838532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.838613 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.838617 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.838683 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.838723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260620 14:34:31.839021 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.835841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54085 (local address 127.0.98.1:45115)
0620 14:34:31.835881 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.835883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.836292 (+ 409us) server_negotiation.cc:407] Connection header received
0620 14:34:31.836321 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.836322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.836336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.836412 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.836491 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.837129 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.838197 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.838646 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.838706 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.838715 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.838895 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.838949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:34:31.839231 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.835765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.98.3:54085)
0620 14:34:31.836229 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.836237 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.836271 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.836411 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.836412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.836413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.836453 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.836454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.837215 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.837216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.838100 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.838103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.838746 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.838747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.838811 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.838816 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.839117 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.839162 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":127,"thread_start_us":121,"threads_started":1}
I20260620 14:34:31.839228 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3964bcbe874eb781f9a5718c29d2bf" candidate_uuid: "33a2585d480b46caa0702b6d18bfb507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f93ed34734317aa3c3f57afc75c92" is_pre_election: true
I20260620 14:34:31.839387 16566 raft_consensus.cc:2454] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a2585d480b46caa0702b6d18bfb507 in term 0.
I20260620 14:34:31.839569 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3964bcbe874eb781f9a5718c29d2bf" candidate_uuid: "33a2585d480b46caa0702b6d18bfb507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495a95a8a2474ca8951a6dc52998a48c" is_pre_election: true
I20260620 14:34:31.839635 16632 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507; no voters:
I20260620 14:34:31.839743 16418 raft_consensus.cc:2454] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a2585d480b46caa0702b6d18bfb507 in term 0.
I20260620 14:34:31.839772 16757 raft_consensus.cc:2781] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:31.839813 16757 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:31.839846 16757 raft_consensus.cc:3037] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.840556 16757 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.840703 16757 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [CANDIDATE]: Term 1 election: Requested vote from peers 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535), 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115)
I20260620 14:34:31.840838 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3964bcbe874eb781f9a5718c29d2bf" candidate_uuid: "33a2585d480b46caa0702b6d18bfb507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495a95a8a2474ca8951a6dc52998a48c"
I20260620 14:34:31.840879 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3964bcbe874eb781f9a5718c29d2bf" candidate_uuid: "33a2585d480b46caa0702b6d18bfb507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022f93ed34734317aa3c3f57afc75c92"
I20260620 14:34:31.840940 16418 raft_consensus.cc:3037] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.840971 16566 raft_consensus.cc:3037] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.841881 16566 raft_consensus.cc:2454] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a2585d480b46caa0702b6d18bfb507 in term 1.
I20260620 14:34:31.842039 16632 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507; no voters:
I20260620 14:34:31.842134 16757 raft_consensus.cc:2781] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:31.842125 16418 raft_consensus.cc:2454] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a2585d480b46caa0702b6d18bfb507 in term 1.
I20260620 14:34:31.842269 16757 raft_consensus.cc:686] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 1 LEADER]: Becoming Leader. State: Replica: 33a2585d480b46caa0702b6d18bfb507, State: Running, Role: LEADER
I20260620 14:34:31.842365 16757 consensus_queue.cc:237] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:31.843155 16286 catalog_manager.cc:5495] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "33a2585d480b46caa0702b6d18bfb507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } health_report { overall_health: UNKNOWN } } }
I20260620 14:34:31.876668 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.874541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.98.3:33711)
0620 14:34:31.874600 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.874602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.874611 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:34:31.874648 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.874651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.874673 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.874732 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:31.874857 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.875305 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.876001 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.876181 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.876235 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.876382 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:31.876477 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:31.876536 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.876576 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.876593 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:34:31.878489 16418 raft_consensus.cc:1261] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Refusing update from remote peer 33a2585d480b46caa0702b6d18bfb507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:31.878527 16566 raft_consensus.cc:1261] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Refusing update from remote peer 33a2585d480b46caa0702b6d18bfb507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:31.878877 16761 consensus_queue.cc:1035] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:31.879048 16757 consensus_queue.cc:1035] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:31.881733 16765 mvcc.cc:205] Tried to move back new op lower bound from 7298933030411644928 to 7298933030267052032. Current Snapshot: MvccSnapshot[applied={T|T < 7298933030411644928}]
I20260620 14:34:31.883664 16285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:31.883775 16285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:31.884539 16285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Sending DeleteTablet for 3 replicas of tablet 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:31.884851 16392 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56742
I20260620 14:34:31.884928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16324
I20260620 14:34:31.884966 16483 tablet_replica.cc:330] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: stopping tablet replica
I20260620 14:34:31.884912 16546 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53626
I20260620 14:34:31.885020 16483 raft_consensus.cc:2229] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:31.885073 16483 raft_consensus.cc:2258] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.887846 16678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3964bcbe874eb781f9a5718c29d2bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53814
I20260620 14:34:31.888109 16774 tablet_replica.cc:330] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: stopping tablet replica
I20260620 14:34:31.888219 16774 raft_consensus.cc:2229] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:31.888343 16774 raft_consensus.cc:2258] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.888723 16774 ts_tablet_manager.cc:1905] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:31.889544 16773 tablet_replica.cc:330] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: stopping tablet replica
I20260620 14:34:31.889699 16773 raft_consensus.cc:2229] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:31.889860 16773 raft_consensus.cc:2258] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.890359 16773 ts_tablet_manager.cc:1905] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:31.890771 16774 ts_tablet_manager.cc:1918] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:31.890847 16774 log.cc:1196] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:31.891116 16774 ts_tablet_manager.cc:1939] T 1b3964bcbe874eb781f9a5718c29d2bf P 33a2585d480b46caa0702b6d18bfb507: Deleting consensus metadata
I20260620 14:34:31.891515 16272 catalog_manager.cc:4841] TS 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): tablet 1b3964bcbe874eb781f9a5718c29d2bf (table test-table [id=f230b7893af54123a5141c1b2d11d3d8]) successfully deleted
W20260620 14:34:31.891937 16273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45115 (error 108)
I20260620 14:34:31.892139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16476
I20260620 14:34:31.892858 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.892727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:41392)
0620 14:34:31.892770 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.892817 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:34:31.892973 16273 catalog_manager.cc:4584] TS 495a95a8a2474ca8951a6dc52998a48c (127.0.98.1:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b3964bcbe874eb781f9a5718c29d2bf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
I20260620 14:34:31.896317 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.896207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.0.1:53632)
0620 14:34:31.896264 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.896289 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:34:31.897089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16615
I20260620 14:34:31.901480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16253
I20260620 14:34:51.906800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--webserver_interface=127.0.98.62
--webserver_port=40107
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:51.985993 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:51.986178 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:51.986199 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:51.987260 16776 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:34:51.987293 16776 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:34:51.987306 16776 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:34:51.987317 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:51.987330 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:51.987344 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:51.987355 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:51.987366 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:51.989343 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:51.989624 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:51.989876 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:51.994836 16782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:51.994918 16776 server_base.cc:1034] running on GCE node
W20260620 14:34:51.994848 16784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:51.994848 16781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:51.995309 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:51.995551 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:51.996723 16776 hybrid_clock.cc:648] HybridClock initialized: now 1781966091996704 us; error 37 us; skew 500 ppm
Jun 20 14:34:51 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:51.999116 16776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:34:51.999583 16776 webserver.cc:460] Webserver started at http://127.0.98.62:40107/ using document root <none> and password file <none>
I20260620 14:34:51.999815 16776 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:51.999886 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:52.001204 16776 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:34:52.001920 16791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.002099 16776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.002197 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "b3654adde630406fb236e50950ff47e5"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "4cb698d6696ca52705107988fe293767"
server_key_iv: "4580232463b1f6a4a26bebbbe4370d78"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.002547 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:52.014523 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:52.016234 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:52.016388 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:52.020509 16776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37501
I20260620 14:34:52.020556 16843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37501 every 8 connection(s)
I20260620 14:34:52.020949 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:52.021504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16776
I20260620 14:34:52.021803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45115
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=35957
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:52.022567 16844 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:34:52.023223 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap starting.
I20260620 14:34:52.031540 16844 log.cc:824] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:52.033854 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:52.034149 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Bootstrap complete.
I20260620 14:34:52.036409 16844 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:52.036643 16844 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3654adde630406fb236e50950ff47e5, State: Initialized, Role: FOLLOWER
I20260620 14:34:52.036867 16844 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:52.036931 16844 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:52.036981 16844 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:52.037029 16844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:34:52.038100 16844 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:52.038183 16844 leader_election.cc:302] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3654adde630406fb236e50950ff47e5; no voters:
I20260620 14:34:52.038393 16844 leader_election.cc:288] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:34:52.038460 16848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:34:52.038614 16848 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [term 3 LEADER]: Becoming Leader. State: Replica: b3654adde630406fb236e50950ff47e5, State: Running, Role: LEADER
I20260620 14:34:52.038664 16844 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:52.038825 16848 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } }
I20260620 14:34:52.039084 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3654adde630406fb236e50950ff47e5. Latest consensus state: current_term: 3 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:52.039250 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b3654adde630406fb236e50950ff47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3654adde630406fb236e50950ff47e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37501 } } }
I20260620 14:34:52.039256 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:52.039402 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:52.039788 16857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:52.040556 16857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:52.040640 16857 catalog_manager.cc:1259] Loaded cluster ID: 7f24aac627044fac95296e009942b30c
I20260620 14:34:52.040663 16857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:52.041966 16857 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:52.042140 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Loaded TSK: 0
I20260620 14:34:52.042183 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3654adde630406fb236e50950ff47e5: Loaded TSK: 1
F20260620 14:34:52.042408 16857 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f931c3179ad google::LogMessage::Fail() at ??:0
@ 0x7f931c31a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f931c3174de google::LogMessage::Flush() at ??:0
@ 0x7f931c318a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f931ea84104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f931ea81d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f93242711cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f932427148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9324271594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9324237c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9324241ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f931e1e50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f931e1da954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f931f03e6db start_thread at ??:0
@ 0x7f931ae1561f clone at ??:0
W20260620 14:34:52.137194 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:52.137524 16846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:52.137566 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:52.139287 16846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:52.139344 16846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:52.139393 16846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:34:52.141530 16846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35957
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:52.141779 16846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:52.142022 16846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:52.147352 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.147379 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:52.147552 16846 server_base.cc:1034] running on GCE node
W20260620 14:34:52.147323 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:52.147912 16846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:52.148191 16846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:52.149359 16846 hybrid_clock.cc:648] HybridClock initialized: now 1781966092149342 us; error 31 us; skew 500 ppm
Jun 20 14:34:52 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:52.151816 16846 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:34:52.152241 16846 webserver.cc:460] Webserver started at http://127.0.98.1:35957/ using document root <none> and password file <none>
I20260620 14:34:52.152472 16846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:52.152541 16846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:52.153990 16846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:52.154733 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.154942 16846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:52.155049 16846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "495a95a8a2474ca8951a6dc52998a48c"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "87d4241a1312513ea6f6602188e839cb"
server_key_iv: "7640eb110aecbb419fe61ed37b3527c6"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.155401 16846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:52.188127 16846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:52.190533 16846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:52.190702 16846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:52.190943 16846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:52.191447 16902 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:34:52.193504 16846 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:34:52.193585 16846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:34:52.193632 16846 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:34:52.194316 16846 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:34:52.194376 16846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:34:52.194463 16902 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap starting.
I20260620 14:34:52.201578 16846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260620 14:34:52.201721 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260620 14:34:52.202014 16846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:52.205202 16902 log.cc:824] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:52.207921 16902 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:52.208225 16902 tablet_bootstrap.cc:492] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Bootstrap complete.
I20260620 14:34:52.209062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 16846
I20260620 14:34:52.209393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39535
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=46851
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:52.209471 16902 ts_tablet_manager.cc:1397] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260620 14:34:52.210883 16902 raft_consensus.cc:348] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:52.211213 16902 raft_consensus.cc:729] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495a95a8a2474ca8951a6dc52998a48c, State: Initialized, Role: FOLLOWER
I20260620 14:34:52.211388 16902 consensus_queue.cc:260] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:52.211664 16902 ts_tablet_manager.cc:1428] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 14:34:52.268700 17012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:34:52.202782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:56999)
0620 14:34:52.202962 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.203005 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:52.203460 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:52.268489 (+ 65029us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260620 14:34:52.268970 17010 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37501 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37501: Client connection negotiation failed: client connection to 127.0.98.62:37501: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:34:52.269330 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.269197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:44881)
0620 14:34:52.269264 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.269301 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:34:52.269701 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.269616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:49539)
0620 14:34:52.269661 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.269684 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:34:52.269819 17010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:34:52.305958 17014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:52.306161 17014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:52.306186 17014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:52.307250 17014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:52.307286 17014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:52.307312 17014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:34:52.308665 17014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46851
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:52.308840 17014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:52.309005 17014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:52.313779 17020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.313769 17023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.313771 17021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:52.314225 17014 server_base.cc:1034] running on GCE node
I20260620 14:34:52.314430 17014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:52.314661 17014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:52.315825 17014 hybrid_clock.cc:648] HybridClock initialized: now 1781966092315813 us; error 28 us; skew 500 ppm
Jun 20 14:34:52 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:52.318228 17014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:34:52.318645 17014 webserver.cc:460] Webserver started at http://127.0.98.2:46851/ using document root <none> and password file <none>
I20260620 14:34:52.318879 17014 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:52.318948 17014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:52.320242 17014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:34:52.320960 17030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.321130 17014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:34:52.321234 17014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "022f93ed34734317aa3c3f57afc75c92"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "99d2d1457c386f0e14a0f54385a74215"
server_key_iv: "8c862579eebf8e3e0d0f213f8e47abca"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.321615 17014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:52.336825 17014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:52.338742 17014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:52.338905 17014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:52.339130 17014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:52.339566 17037 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 14:34:52.340922 17037 ts_tablet_manager.cc:1852] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 14:34:52.341183 17037 ts_tablet_manager.cc:1905] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:52.343446 17037 ts_tablet_manager.cc:1918] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:52.343534 17037 log.cc:1196] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:52.343852 17037 ts_tablet_manager.cc:1939] T 1b3964bcbe874eb781f9a5718c29d2bf P 022f93ed34734317aa3c3f57afc75c92: Deleting consensus metadata
I20260620 14:34:52.344067 17014 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 14:34:52.344141 17014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 14:34:52.344203 17014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:52.344235 17014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.350569 17014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39535
I20260620 14:34:52.350620 17144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39535 every 8 connection(s)
I20260620 14:34:52.350937 17014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:52.351923 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.351570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:50695)
0620 14:34:52.351726 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.351791 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260620 14:34:52.352150 17145 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37501 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37501: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
I20260620 14:34:52.352458 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.352375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:36561)
0620 14:34:52.352418 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.352442 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:34:52.352732 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.352663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:57553)
0620 14:34:52.352700 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.352720 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:34:52.352860 17145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:34:52.355324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17014
I20260620 14:34:52.355549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33711
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37501
--webserver_port=44711
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:52.435407 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:52.435604 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:52.435626 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:52.436851 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:52.436903 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:52.436930 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:34:52.438398 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44711
--tserver_master_addrs=127.0.98.62:37501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:34:52.438640 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:52.438879 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:52.443791 17154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.443846 17153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.443796 17156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:52.444073 17148 server_base.cc:1034] running on GCE node
I20260620 14:34:52.444285 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:52.444519 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:52.445695 17148 hybrid_clock.cc:648] HybridClock initialized: now 1781966092445686 us; error 41 us; skew 500 ppm
Jun 20 14:34:52 dist-test-slave-tc2g krb5kdc[15706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:52.448101 17148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:34:52.448601 17148 webserver.cc:460] Webserver started at http://127.0.98.3:44711/ using document root <none> and password file <none>
I20260620 14:34:52.448824 17148 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:52.448892 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:52.450284 17148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:34:52.450861 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.451033 17148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:34:52.451133 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "33a2585d480b46caa0702b6d18bfb507"
format_stamp: "Formatted at 2026-06-20 14:34:10 on dist-test-slave-tc2g"
server_key: "a1f7c848cc6ef526cceb8fa03a25fdfa"
server_key_iv: "58d2a01fbb55580fd5a0de3a25288b04"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.451478 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:52.466738 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:52.468616 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:52.468780 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:52.469018 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:52.469326 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:52.469378 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.469413 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:52.469488 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.475507 17148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33711
I20260620 14:34:52.475559 17276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33711 every 8 connection(s)
I20260620 14:34:52.475875 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:52.476863 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.476506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:51549)
0620 14:34:52.476657 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.476725 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
W20260620 14:34:52.477269 17277 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37501 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37501: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
I20260620 14:34:52.477677 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.477548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:44255)
0620 14:34:52.477625 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.477657 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:34:52.477990 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.477931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:52737)
0620 14:34:52.477954 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:52.477975 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 14:34:52.478102 17277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:34:52.480268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17148
I20260620 14:34:53.270427 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.270210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:47789)
0620 14:34:53.270334 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:53.270369 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260620 14:34:53.353375 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.353168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:43173)
0620 14:34:53.353281 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:53.353324 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260620 14:34:53.480387 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.478447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:38445)
0620 14:34:53.480266 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:53.480295 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":75,"threads_started":1}
I20260620 14:34:53.636091 17284 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:53.636220 17284 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:53.636615 17284 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
Jun 20 14:34:53 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 14:34:53 dist-test-slave-tc2g krb5kdc[15706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 14:34:53.647171 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.636947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51953 (local address 127.0.98.2:39535)
0620 14:34:53.637187 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:53.637195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:53.637216 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:34:53.637835 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:53.637838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:53.637923 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:53.638068 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:53.638677 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.639529 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.640556 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.640968 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.641099 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.643565 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:53.643570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:53.643570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:53.643582 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:53.645492 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.645871 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.645875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.645875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.645986 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.646266 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.646267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.646267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.646511 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:53.646604 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:53.646916 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:34:53.647054 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":60,"threads_started":1}
I20260620 14:34:53.647185 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.636821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.98.1:51953)
0620 14:34:53.636862 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:53.636873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:53.637087 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:53.638073 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:53.638074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:53.638365 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:53.638557 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:53.638562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.639644 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.639645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:53.640462 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:53.640464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.641067 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.641068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:53.641183 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.641849 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:53.641861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:53.643436 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:53.645568 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:53.645574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:53.645576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:53.645788 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:53.646086 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:53.646087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:53.646088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:53.646196 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:53.646587 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:53.646594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:53.646728 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:34:53.646848 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:34:53.647050 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":7}
I20260620 14:34:53.647318 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.636930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55629 (local address 127.0.98.3:33711)
0620 14:34:53.637143 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:53.637156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:53.637197 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:34:53.637737 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:53.637740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:53.637784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:53.637903 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:53.638679 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.639545 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.640587 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.640968 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.641093 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.643932 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:53.643935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:53.643936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:53.643946 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:53.645870 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.646170 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.646174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.646174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.646273 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.646496 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.646497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.646497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.646726 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:53.646829 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:53.647051 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:34:53.647184 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
I20260620 14:34:53.647866 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.636821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33711 (local address 127.0.98.1:55629)
0620 14:34:53.637004 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:53.637013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:53.637072 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:53.637979 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:53.637982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:53.638399 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:53.638557 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:53.638562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.639719 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.639720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:53.640462 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:53.640465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.641065 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.641066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:53.641184 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.641790 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:53.641799 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:53.643822 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:53.645937 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:53.645937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:53.645937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:53.646069 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:53.646344 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:53.646345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:53.646345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:53.646427 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:53.646812 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:53.646813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:53.646863 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:34:53.646990 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:34:53.647116 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":134,"thread_start_us":69,"threads_started":1}
W20260620 14:34:53.648012 16897 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
W20260620 14:34:53.648562 16898 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:53.648607 16898 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 495a95a8a2474ca8951a6dc52998a48c; no voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507
I20260620 14:34:53.648723 17284 raft_consensus.cc:2726] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:54.271130 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:54.270915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:55695)
0620 14:34:54.271036 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:54.271073 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260620 14:34:54.354162 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:54.353974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:44639)
0620 14:34:54.354094 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:54.354128 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260620 14:34:54.481063 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:54.480883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:45027)
0620 14:34:54.480991 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:54.481023 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260620 14:34:55.271772 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:55.271570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:58527)
0620 14:34:55.271689 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:55.271726 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 14:34:55.354766 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:55.354555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:38529)
0620 14:34:55.354682 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:55.354718 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260620 14:34:55.481702 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:55.481526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:44313)
0620 14:34:55.481635 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:55.481669 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260620 14:34:55.798887 17294 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:55.798982 17294 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:55.799201 17294 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
W20260620 14:34:55.799644 16898 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
W20260620 14:34:55.799721 16897 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:55.799770 16897 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 495a95a8a2474ca8951a6dc52998a48c; no voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507
I20260620 14:34:55.799868 17294 raft_consensus.cc:2726] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:56.272449 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:56.272229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:45477)
0620 14:34:56.272358 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:56.272400 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260620 14:34:56.355392 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:56.355208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:54019)
0620 14:34:56.355318 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:56.355354 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260620 14:34:56.482270 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:56.482094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:49793)
0620 14:34:56.482202 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:56.482236 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260620 14:34:57.273126 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:57.272902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:41515)
0620 14:34:57.273046 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:57.273080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260620 14:34:57.355932 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:57.355755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:37949)
0620 14:34:57.355861 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:57.355895 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260620 14:34:57.482810 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:57.482645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:59303)
0620 14:34:57.482741 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:57.482773 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":58,"threads_started":1}
I20260620 14:34:57.779584 17301 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:57.779676 17301 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:34:57.779881 17301 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
W20260620 14:34:57.780200 16898 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
W20260620 14:34:57.780277 16897 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:34:57.780359 16897 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 495a95a8a2474ca8951a6dc52998a48c; no voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507
I20260620 14:34:57.780424 17301 raft_consensus.cc:2726] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:58.273829 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:58.273613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:41291)
0620 14:34:58.273733 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:58.273773 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 14:34:58.356519 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:58.356324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:36289)
0620 14:34:58.356437 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:58.356479 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260620 14:34:58.483425 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:58.483163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:58765)
0620 14:34:58.483352 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:58.483392 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":55,"threads_started":1}
I20260620 14:34:59.274492 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:59.274294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:48789)
0620 14:34:59.274410 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:59.274445 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260620 14:34:59.357113 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:59.356921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:54409)
0620 14:34:59.357037 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:59.357072 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260620 14:34:59.484228 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:59.483839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:57023)
0620 14:34:59.483955 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:59.484183 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260620 14:35:00.275079 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:00.274890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:36903)
0620 14:35:00.275005 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:00.275033 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 14:35:00.357837 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:00.357581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:60489)
0620 14:35:00.357682 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:00.357742 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260620 14:35:00.484884 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:00.484711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:45725)
0620 14:35:00.484819 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:00.484848 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260620 14:35:00.844627 17311 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:00.844717 17311 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:35:00.844919 17311 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
W20260620 14:35:00.845274 16897 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
W20260620 14:35:00.845381 16898 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:35:00.845436 16898 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 495a95a8a2474ca8951a6dc52998a48c; no voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507
I20260620 14:35:00.845543 17311 raft_consensus.cc:2726] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:35:01.275753 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.275547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:49197)
0620 14:35:01.275670 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.275706 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260620 14:35:01.358436 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.358244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:34159)
0620 14:35:01.358363 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.358397 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260620 14:35:01.485558 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.485324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:56245)
0620 14:35:01.485484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.485521 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260620 14:35:02.276475 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.276268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:34801)
0620 14:35:02.276395 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.276429 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260620 14:35:02.359064 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.358847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:37329)
0620 14:35:02.358989 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.359024 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260620 14:35:02.486162 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.485960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:48435)
0620 14:35:02.486093 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.486126 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260620 14:35:03.080602 17318 raft_consensus.cc:482] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:03.080698 17318 raft_consensus.cc:504] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a2585d480b46caa0702b6d18bfb507" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33711 } } peers { permanent_uuid: "022f93ed34734317aa3c3f57afc75c92" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "495a95a8a2474ca8951a6dc52998a48c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } }
I20260620 14:35:03.080911 17318 leader_election.cc:288] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711), 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535)
W20260620 14:35:03.081292 16897 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 022f93ed34734317aa3c3f57afc75c92 (127.0.98.2:39535): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
W20260620 14:35:03.081368 16898 leader_election.cc:341] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33a2585d480b46caa0702b6d18bfb507 (127.0.98.3:33711): Not found: Tablet not found: 1b3964bcbe874eb781f9a5718c29d2bf
I20260620 14:35:03.081451 16898 leader_election.cc:302] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 495a95a8a2474ca8951a6dc52998a48c; no voters: 022f93ed34734317aa3c3f57afc75c92, 33a2585d480b46caa0702b6d18bfb507
I20260620 14:35:03.081550 17318 raft_consensus.cc:2726] T 1b3964bcbe874eb781f9a5718c29d2bf P 495a95a8a2474ca8951a6dc52998a48c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:35:03.277141 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.276889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:38027)
0620 14:35:03.277054 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:03.277094 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260620 14:35:03.359601 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.359423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:51955)
0620 14:35:03.359528 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:03.359557 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260620 14:35:03.486734 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.486557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:52145)
0620 14:35:03.486663 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:03.486697 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260620 14:35:04.277786 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.277590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:35743)
0620 14:35:04.277707 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.277739 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260620 14:35:04.360256 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.360023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:53247)
0620 14:35:04.360158 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.360213 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260620 14:35:04.487250 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.487072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:44851)
0620 14:35:04.487183 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.487213 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260620 14:35:05.278451 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.278217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:58121)
0620 14:35:05.278369 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.278405 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260620 14:35:05.360973 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.360763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:60985)
0620 14:35:05.360897 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.360933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 14:35:05.487897 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.487694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:38965)
0620 14:35:05.487814 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.487848 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 14:35:06.279107 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.278897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:49711)
0620 14:35:06.279019 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:06.279056 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260620 14:35:06.361649 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.361345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:53961)
0620 14:35:06.361571 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:06.361606 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":57,"threads_started":1}
I20260620 14:35:06.488600 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.488384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:49621)
0620 14:35:06.488525 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:06.488561 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260620 14:35:07.279831 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:07.279635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.1:43805)
0620 14:35:07.279753 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:07.279785 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260620 14:35:07.362365 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:07.362122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.2:52469)
0620 14:35:07.362253 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:07.362290 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 14:35:07.487993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 16846
I20260620 14:35:07.489593 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:07.489357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37501 (local address 127.0.98.3:42445)
0620 14:35:07.489490 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:07.489536 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 14:35:07.492852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17014
I20260620 14:35:07.497190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17148
2026-06-20T14:35:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58282 ms)
[----------] 30 tests from SecurityITest (230597 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:07 dist-test-slave-tc2g krb5kdc[17336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:07 dist-test-slave-tc2g krb5kdc[17336](info): set up 2 sockets
Jun 20 14:35:07 dist-test-slave-tc2g krb5kdc[17336](info): commencing operation
krb5kdc: starting...
W20260620 14:35:08.533604 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:08 dist-test-slave-tc2g krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:08.648707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:08.728657 17349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:08.728866 17349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:08.728888 17349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:08.730084 17349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:08.730127 17349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:08.730142 17349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:08.730195 17349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:08.730219 17349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:08.731674 17349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:08.731930 17349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:08.732159 17349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:08.736815 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:08.736842 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:08.736831 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:08.737142 17349 server_base.cc:1034] running on GCE node
I20260620 14:35:08.737391 17349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:08.737720 17349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:08.738894 17349 hybrid_clock.cc:648] HybridClock initialized: now 1781966108738886 us; error 39 us; skew 500 ppm
Jun 20 14:35:08 dist-test-slave-tc2g krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:08.741286 17349 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:08.741755 17349 webserver.cc:460] Webserver started at http://127.0.98.62:41529/ using document root <none> and password file <none>
I20260620 14:35:08.741950 17349 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:08.741992 17349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:08.742076 17349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:08.743093 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "e246da25e07c4559a4880cde31179670"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "bb199d7b727c9ebf838b4cb0c466e7f9"
server_key_iv: "3e9bc3a8c903e21c81bbc12b9d6c06d4"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.743396 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "e246da25e07c4559a4880cde31179670"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "bb199d7b727c9ebf838b4cb0c466e7f9"
server_key_iv: "3e9bc3a8c903e21c81bbc12b9d6c06d4"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.744596 17349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:08.745223 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:08.745395 17349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:08.745528 17349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "e246da25e07c4559a4880cde31179670"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "bb199d7b727c9ebf838b4cb0c466e7f9"
server_key_iv: "3e9bc3a8c903e21c81bbc12b9d6c06d4"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.745584 17349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:08.761933 17349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:08.763713 17349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:08.763824 17349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:08.767647 17349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40601
I20260620 14:35:08.767704 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40601 every 8 connection(s)
I20260620 14:35:08.768036 17349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:08.768913 17417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:08.771135 17417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: Bootstrap starting.
I20260620 14:35:08.771651 17417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:08.771878 17417 log.cc:824] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:08.772519 17417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: No bootstrap required, opened a new log
I20260620 14:35:08.773331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17349
I20260620 14:35:08.773468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:08.773675 392 external_mini_cluster.cc:1428] Setting key 9133b7515856b495a9a1669aee4ccdd3
I20260620 14:35:08.774665 17417 raft_consensus.cc:348] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:35:08.774834 17417 raft_consensus.cc:374] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:08.774876 17417 raft_consensus.cc:729] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e246da25e07c4559a4880cde31179670, State: Initialized, Role: FOLLOWER
I20260620 14:35:08.775014 17417 consensus_queue.cc:260] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:35:08.775102 17417 raft_consensus.cc:388] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:08.775141 17417 raft_consensus.cc:482] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:08.775195 17417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:08.776165 17417 raft_consensus.cc:504] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:35:08.776288 17417 leader_election.cc:302] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e246da25e07c4559a4880cde31179670; no voters:
I20260620 14:35:08.776464 17417 leader_election.cc:288] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:08.776507 17422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:08.776678 17422 raft_consensus.cc:686] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [term 1 LEADER]: Becoming Leader. State: Replica: e246da25e07c4559a4880cde31179670, State: Running, Role: LEADER
I20260620 14:35:08.776714 17417 sys_catalog.cc:564] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:08.776839 17422 consensus_queue.cc:237] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } }
I20260620 14:35:08.777285 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e246da25e07c4559a4880cde31179670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:35:08.777359 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:08.777310 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e246da25e07c4559a4880cde31179670. Latest consensus state: current_term: 1 leader_uuid: "e246da25e07c4559a4880cde31179670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e246da25e07c4559a4880cde31179670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40601 } } }
I20260620 14:35:08.777410 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:08.777673 17431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:08.778266 17431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:08.779733 17431 catalog_manager.cc:1347] Generated new cluster ID: ed4e512d45ee465e97d875e3370d7ec4
I20260620 14:35:08.780220 17431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:35:08 dist-test-slave-tc2g krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:08.785302 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:08.774167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.98.62:40601)
0620 14:35:08.774342 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:08.774361 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:08.774385 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:35:08.774644 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:08.774649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:08.774693 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:08.774800 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:08.775603 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.776658 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.777754 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.778374 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.778863 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:08.781185 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:08.781189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:08.781189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:08.781200 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:08.783112 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:08.783478 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:08.783481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:08.783481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:08.783964 (+ 483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:08.784162 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:08.784163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:08.784163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:08.784439 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:08.784912 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:08.784977 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:35:08.785100 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260620 14:35:08.807410 17431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:08.807947 17431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:08.825390 17431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: Generated new TSK 0
I20260620 14:35:08.825628 17431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:35:08.860152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:08.940604 17443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:08.940800 17443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:08.940824 17443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:08.942039 17443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:08.942078 17443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:08.942128 17443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:08.943652 17443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:08.943887 17443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:08.944101 17443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:08.949000 17448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:08.948964 17449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:08.949016 17443 server_base.cc:1034] running on GCE node
W20260620 14:35:08.948982 17451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:08.949522 17443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:08.949764 17443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:08.950923 17443 hybrid_clock.cc:648] HybridClock initialized: now 1781966108950925 us; error 39 us; skew 500 ppm
Jun 20 14:35:08 dist-test-slave-tc2g krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:08.953534 17443 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:35:08.953941 17443 webserver.cc:460] Webserver started at http://127.0.98.1:38533/ using document root <none> and password file <none>
I20260620 14:35:08.954174 17443 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:08.954242 17443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:08.954421 17443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:08.955441 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab993a66f03d4d808e5e14bb9a70578e"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "fa48ee93841180d576de375c164b32bc"
server_key_iv: "04560f58c6f8b951eccc4b31a4933ddb"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.955796 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab993a66f03d4d808e5e14bb9a70578e"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "fa48ee93841180d576de375c164b32bc"
server_key_iv: "04560f58c6f8b951eccc4b31a4933ddb"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.957062 17443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:08.957834 17458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:08.958012 17443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:08.958110 17443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "ab993a66f03d4d808e5e14bb9a70578e"
format_stamp: "Formatted at 2026-06-20 14:35:08 on dist-test-slave-tc2g"
server_key: "fa48ee93841180d576de375c164b32bc"
server_key_iv: "04560f58c6f8b951eccc4b31a4933ddb"
server_key_version: "encryptionkey@0"
I20260620 14:35:08.958192 17443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:08.980370 17443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:08.982800 17443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:08.982972 17443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:08.983207 17443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:08.983531 17443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:08.983585 17443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:08.983628 17443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:08.983655 17443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:08.989526 17443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35641
I20260620 14:35:08.989571 17571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35641 every 8 connection(s)
I20260620 14:35:08.989921 17443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:35:08.994997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17443
I20260620 14:35:08.995119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:08.995335 392 external_mini_cluster.cc:1428] Setting key d062c4b9ae3baaff5cf41d763c611896
Jun 20 14:35:08 dist-test-slave-tc2g krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:08.999554 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:08.990649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41339 (local address 127.0.98.62:40601)
0620 14:35:08.990715 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:08.990719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:08.990928 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:35:08.991857 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:08.991858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:08.991871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:08.991932 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:08.992514 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.993022 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.994089 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.994291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.994345 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:08.996688 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:08.996690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:08.996690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:08.996701 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:08.998156 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:08.998589 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:08.998590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:08.998590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:08.998647 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:08.998903 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:08.998904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:08.998904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:08.999016 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:08.999091 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:08.999372 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:35:08.999461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:35:08.999784 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:08.990635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.1:41339)
0620 14:35:08.990800 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:08.990842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:08.991762 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:08.991986 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:08.991989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:08.992235 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:08.992426 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:08.992430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.993100 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.993101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:08.994011 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:08.994014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:08.994363 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:08.994364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:08.994462 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:08.994974 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:08.994985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:08.996586 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:08.998262 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:08.998269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:08.998272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:08.998502 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:08.998722 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:08.998723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:08.998723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:08.998834 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:08.999084 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:08.999090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:08.999230 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 14:35:08.999350 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:35:08.999509 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 14:35:09.000288 17572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:35:09.000404 17572 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:09.000620 17572 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:35:09.001228 17381 ts_manager.cc:194] Registered new tserver with Master: ab993a66f03d4d808e5e14bb9a70578e (127.0.98.1:35641)
I20260620 14:35:09.002218 17381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41339
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:35:09.019716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:09.098428 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:09.098634 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:09.098659 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:09.099750 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:09.099792 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:09.099819 17577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:09.101281 17577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:09.101536 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:09.101725 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:09.106606 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.106627 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.106612 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.106916 17577 server_base.cc:1034] running on GCE node
I20260620 14:35:09.107151 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:09.107385 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:09.108563 17577 hybrid_clock.cc:648] HybridClock initialized: now 1781966109108549 us; error 44 us; skew 500 ppm
Jun 20 14:35:09 dist-test-slave-tc2g krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:09.110978 17577 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:35:09.111454 17577 webserver.cc:460] Webserver started at http://127.0.98.2:39231/ using document root <none> and password file <none>
I20260620 14:35:09.111670 17577 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:09.111738 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:09.111886 17577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:09.112912 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "86979f16c49c4e05bc536fb29cb3098a"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "44b46a328626c0f7726f9f4c33cd75ad"
server_key_iv: "64a7b0ef500cca828f801ef8f41f3c63"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.113274 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86979f16c49c4e05bc536fb29cb3098a"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "44b46a328626c0f7726f9f4c33cd75ad"
server_key_iv: "64a7b0ef500cca828f801ef8f41f3c63"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.114566 17577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:09.115237 17592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.115417 17577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:09.115515 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "86979f16c49c4e05bc536fb29cb3098a"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "44b46a328626c0f7726f9f4c33cd75ad"
server_key_iv: "64a7b0ef500cca828f801ef8f41f3c63"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.115595 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:09.135054 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:09.137465 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:09.137633 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:09.137868 17577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:09.138180 17577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:09.138233 17577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.138274 17577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:09.138302 17577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.144163 17577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33711
I20260620 14:35:09.144214 17705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33711 every 8 connection(s)
I20260620 14:35:09.144522 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:35:09 dist-test-slave-tc2g krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:09.153208 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.145204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47531 (local address 127.0.98.62:40601)
0620 14:35:09.145248 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.145250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.145471 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:35:09.146352 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.146353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.146364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.146443 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.146982 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.147575 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.148581 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.148766 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.148819 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.150900 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.150901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.150902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:09.150908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.152098 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.152456 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.152456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.152457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.152505 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.152704 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.152704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.152704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.152796 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.152850 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.153067 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.153132 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:35:09.153362 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.145166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.2:47531)
0620 14:35:09.145311 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.145381 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.146261 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.146493 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.146496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.146730 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:09.146910 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.146914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.147654 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.147655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.148513 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.148515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.148835 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.148836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.148952 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.149487 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:09.149496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:09.150819 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:09.152171 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:09.152176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:09.152178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:09.152385 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:09.152561 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:09.152561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:09.152562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:09.152653 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:09.152848 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:09.152851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:09.152959 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.153045 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.153170 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260620 14:35:09.153918 17706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:35:09.154024 17706 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:09.154196 17706 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:35:09.154284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17577
I20260620 14:35:09.154425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:09.154597 17381 ts_manager.cc:194] Registered new tserver with Master: 86979f16c49c4e05bc536fb29cb3098a (127.0.98.2:33711)
I20260620 14:35:09.154662 392 external_mini_cluster.cc:1428] Setting key 6e9e4018ac0ceadd5845b56619e75f87
I20260620 14:35:09.155265 17381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47531
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:35:09.178215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:09.258755 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:09.259097 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:09.259234 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:09.260501 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:09.260560 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:09.260600 17711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:09.262603 17711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:09.262791 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:09.262966 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:09.268042 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.268039 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.268038 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.268276 17711 server_base.cc:1034] running on GCE node
I20260620 14:35:09.268512 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:09.268752 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:09.269910 17711 hybrid_clock.cc:648] HybridClock initialized: now 1781966109269900 us; error 33 us; skew 500 ppm
Jun 20 14:35:09 dist-test-slave-tc2g krb5kdc[17336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:09.272413 17711 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:35:09.272822 17711 webserver.cc:460] Webserver started at http://127.0.98.3:40351/ using document root <none> and password file <none>
I20260620 14:35:09.273042 17711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:09.273111 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:09.273270 17711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:09.274354 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "8be028979215449b908f14d0164fda1c"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f88193fe04cffea78caa72ec1909ab80"
server_key_iv: "b635a409e05d00ae997b3feef6ef24a4"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.274724 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8be028979215449b908f14d0164fda1c"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f88193fe04cffea78caa72ec1909ab80"
server_key_iv: "b635a409e05d00ae997b3feef6ef24a4"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.276038 17711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:09.276739 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.276908 17711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:09.277004 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "8be028979215449b908f14d0164fda1c"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f88193fe04cffea78caa72ec1909ab80"
server_key_iv: "b635a409e05d00ae997b3feef6ef24a4"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.277091 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:09.287989 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:09.290278 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:09.290439 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:09.290660 17711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:09.290952 17711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:09.291006 17711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.291046 17711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:09.291074 17711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.296880 17711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43193
I20260620 14:35:09.296936 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43193 every 8 connection(s)
I20260620 14:35:09.297216 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:09.302862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17711
I20260620 14:35:09.302958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
Jun 20 14:35:09 dist-test-slave-tc2g krb5kdc[17336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:09.303145 392 external_mini_cluster.cc:1428] Setting key d2abb9d42ee5d48da68058c6332381aa
I20260620 14:35:09.306057 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.297886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36039 (local address 127.0.98.62:40601)
0620 14:35:09.297927 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.297929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.298108 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:35:09.299020 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.299021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.299030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.299090 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.299681 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.300161 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.301103 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.301284 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.301337 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.303588 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.303589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.303590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:09.303598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.304867 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.305230 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.305230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.305231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.305281 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.305557 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.305558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.305558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.305659 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.305708 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.305936 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.305997 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:35:09.306231 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.297820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40601 (local address 127.0.98.3:36039)
0620 14:35:09.297982 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.298022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.298909 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.299166 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.299169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.299402 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:09.299607 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.299611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.300245 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.300246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.301027 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.301030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.301356 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.301356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.301507 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.301925 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:09.301933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:09.303493 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:09.304933 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:09.304938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:09.304940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:09.305162 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:09.305338 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:09.305338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:09.305339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:09.305495 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:09.305711 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:09.305715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:09.305822 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.305905 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.306015 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260620 14:35:09.306695 17840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40601
I20260620 14:35:09.306767 17840 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:09.306942 17840 heartbeater.cc:511] Master 127.0.98.62:40601 requested a full tablet report, sending...
I20260620 14:35:09.307299 17380 ts_manager.cc:194] Registered new tserver with Master: 8be028979215449b908f14d0164fda1c (127.0.98.3:43193)
I20260620 14:35:09.307667 17380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36039
I20260620 14:35:09.314818 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:09.321826 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.316118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.98.62:40601)
0620 14:35:09.316170 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.316172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.316262 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:35:09.316303 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.316303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.316312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.316360 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.316866 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.317320 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.318115 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.318339 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.318392 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.319589 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.319590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.319590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:09.319595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.320727 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.321022 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.321023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.321023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.321069 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:09.321218 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:09.321219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:09.321219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:09.321372 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.321494 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.321650 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.321729 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:35:09.323588 17380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:09.325243 17380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:09.333015 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.328963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35641 (local address 127.0.0.1:46970)
0620 14:35:09.329304 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.329345 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.329487 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.329716 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.329719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.329721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:09.329791 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.329793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.331056 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.331057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.331957 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.331960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.332729 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.332730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.332812 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.332828 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.332892 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.332938 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":100,"thread_start_us":47,"threads_started":1}
I20260620 14:35:09.333355 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.329082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.98.1:35641)
0620 14:35:09.329273 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.329278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.329468 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:35:09.329568 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.329570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.329603 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.329703 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:09.329851 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.330959 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.332051 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.332639 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.332717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.332776 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.333209 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.333265 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260620 14:35:09.333657 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.328963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43193 (local address 127.0.0.1:39134)
0620 14:35:09.329303 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.329345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.329417 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.329924 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.329926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.329949 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:09.329985 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.329986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.331175 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.331176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.332379 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.332382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.333362 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.333363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.333462 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.333467 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.333527 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.333566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":26,"thread_start_us":50,"threads_started":1}
I20260620 14:35:09.333745 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.329176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33711 (local address 127.0.0.1:51088)
0620 14:35:09.329545 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.329553 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.329597 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.329830 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.329831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.329834 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:09.329888 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.329889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.330959 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.330960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.332379 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.332382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.333373 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.333374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.333590 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.333593 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.333642 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.333678 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":85,"thread_start_us":65,"threads_started":1}
I20260620 14:35:09.333951 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.329292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.0.98.2:33711)
0620 14:35:09.329466 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.329470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.329613 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:35:09.329656 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.329658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.329673 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.329744 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:09.329979 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.330856 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.332479 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.333299 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.333361 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.333438 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.333831 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.333876 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260620 14:35:09.334069 17506 tablet_service.cc:1467] Processing CreateTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 (DEFAULT_TABLE table=test-table [id=232e1e0e0ead4f9c8323946e3a715175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:09.334434 17506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fd4fc739e34d1fb885e66fe7018991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:09.337030 17857 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Bootstrap starting.
I20260620 14:35:09.337381 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.329067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.0.98.3:43193)
0620 14:35:09.329353 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.329358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.329653 (+ 295us) server_negotiation.cc:407] Connection header received
0620 14:35:09.329717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.329720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.329735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.329838 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:09.330049 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.331091 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.332469 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.333216 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.333277 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.333329 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.333563 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.333608 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
I20260620 14:35:09.337733 17857 tablet_bootstrap.cc:654] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:09.337703 17640 tablet_service.cc:1467] Processing CreateTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 (DEFAULT_TABLE table=test-table [id=232e1e0e0ead4f9c8323946e3a715175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:09.338032 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fd4fc739e34d1fb885e66fe7018991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:09.338035 17774 tablet_service.cc:1467] Processing CreateTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 (DEFAULT_TABLE table=test-table [id=232e1e0e0ead4f9c8323946e3a715175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:09.338349 17857 log.cc:824] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:09.338390 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fd4fc739e34d1fb885e66fe7018991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:09.339187 17857 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: No bootstrap required, opened a new log
I20260620 14:35:09.339257 17857 ts_tablet_manager.cc:1397] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:09.340960 17859 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Bootstrap starting.
I20260620 14:35:09.341310 17857 raft_consensus.cc:348] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.341516 17857 raft_consensus.cc:374] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:09.341563 17857 raft_consensus.cc:729] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab993a66f03d4d808e5e14bb9a70578e, State: Initialized, Role: FOLLOWER
I20260620 14:35:09.341575 17859 tablet_bootstrap.cc:654] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:09.341691 17857 consensus_queue.cc:260] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.341809 17860 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Bootstrap starting.
I20260620 14:35:09.341979 17857 ts_tablet_manager.cc:1428] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:09.342015 17572 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:35:09.342114 17859 log.cc:824] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:09.342468 17860 tablet_bootstrap.cc:654] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:09.342880 17859 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: No bootstrap required, opened a new log
I20260620 14:35:09.342967 17859 ts_tablet_manager.cc:1397] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:35:09.343030 17860 log.cc:824] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:09.343794 17860 tablet_bootstrap.cc:492] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: No bootstrap required, opened a new log
I20260620 14:35:09.343864 17860 ts_tablet_manager.cc:1397] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:09.344960 17859 raft_consensus.cc:348] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.345136 17859 raft_consensus.cc:374] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:09.345172 17859 raft_consensus.cc:729] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86979f16c49c4e05bc536fb29cb3098a, State: Initialized, Role: FOLLOWER
I20260620 14:35:09.345279 17859 consensus_queue.cc:260] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.345597 17859 ts_tablet_manager.cc:1428] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:35:09.345605 17706 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
I20260620 14:35:09.345822 17860 raft_consensus.cc:348] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.346019 17860 raft_consensus.cc:374] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:09.346065 17860 raft_consensus.cc:729] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be028979215449b908f14d0164fda1c, State: Initialized, Role: FOLLOWER
I20260620 14:35:09.346169 17860 consensus_queue.cc:260] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.346446 17860 ts_tablet_manager.cc:1428] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:09.346465 17840 heartbeater.cc:503] Master 127.0.98.62:40601 was elected leader, sending a full tablet report...
W20260620 14:35:09.395161 17707 tablet.cc:2369] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:09.462708 17865 raft_consensus.cc:482] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:09.462836 17865 raft_consensus.cc:504] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.463300 17865 leader_election.cc:288] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab993a66f03d4d808e5e14bb9a70578e (127.0.98.1:35641), 86979f16c49c4e05bc536fb29cb3098a (127.0.98.2:33711)
I20260620 14:35:09.467183 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.463578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59315 (local address 127.0.98.1:35641)
0620 14:35:09.463622 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.463624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.463635 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:09.463711 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.463713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.463727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.463801 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:09.463973 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.464736 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466265 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.466813 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466908 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.466918 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.467075 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.467110 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:09.467183 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.463615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33455 (local address 127.0.98.2:33711)
0620 14:35:09.463684 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.463687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.463913 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:35:09.463974 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.463975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.463987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.464044 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:09.464172 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.464840 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466260 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.466813 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.466911 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.467076 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.467116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:09.467202 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.463503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35641 (local address 127.0.98.3:59315)
0620 14:35:09.463564 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.463576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.463628 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.463812 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.463813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.463815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:09.463892 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.463894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.464823 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.464823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.466172 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.466174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.466918 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.466997 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.466999 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.467056 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.467108 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":55}
I20260620 14:35:09.467209 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.463511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33711 (local address 127.0.98.3:33455)
0620 14:35:09.463838 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.463848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.463916 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.464055 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.464056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.464057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:09.464110 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.464111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.464925 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.464926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.466172 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.466175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.466926 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.466926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.466993 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.466997 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.467056 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.467107 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":164,"thread_start_us":105,"threads_started":1}
I20260620 14:35:09.467551 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fd4fc739e34d1fb885e66fe7018991" candidate_uuid: "8be028979215449b908f14d0164fda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab993a66f03d4d808e5e14bb9a70578e" is_pre_election: true
I20260620 14:35:09.467713 17526 raft_consensus.cc:2454] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be028979215449b908f14d0164fda1c in term 0.
I20260620 14:35:09.467911 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fd4fc739e34d1fb885e66fe7018991" candidate_uuid: "8be028979215449b908f14d0164fda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86979f16c49c4e05bc536fb29cb3098a" is_pre_election: true
I20260620 14:35:09.468022 17660 raft_consensus.cc:2454] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be028979215449b908f14d0164fda1c in term 0.
I20260620 14:35:09.468250 17727 leader_election.cc:302] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be028979215449b908f14d0164fda1c, ab993a66f03d4d808e5e14bb9a70578e; no voters:
I20260620 14:35:09.468389 17865 raft_consensus.cc:2781] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:09.468446 17865 raft_consensus.cc:482] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:09.468473 17865 raft_consensus.cc:3037] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:09.469177 17865 raft_consensus.cc:504] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.469319 17865 leader_election.cc:288] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [CANDIDATE]: Term 1 election: Requested vote from peers ab993a66f03d4d808e5e14bb9a70578e (127.0.98.1:35641), 86979f16c49c4e05bc536fb29cb3098a (127.0.98.2:33711)
I20260620 14:35:09.469496 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fd4fc739e34d1fb885e66fe7018991" candidate_uuid: "8be028979215449b908f14d0164fda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86979f16c49c4e05bc536fb29cb3098a"
I20260620 14:35:09.469529 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fd4fc739e34d1fb885e66fe7018991" candidate_uuid: "8be028979215449b908f14d0164fda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab993a66f03d4d808e5e14bb9a70578e"
I20260620 14:35:09.469604 17526 raft_consensus.cc:3037] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:09.469605 17660 raft_consensus.cc:3037] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:09.470299 17660 raft_consensus.cc:2454] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be028979215449b908f14d0164fda1c in term 1.
I20260620 14:35:09.470355 17526 raft_consensus.cc:2454] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be028979215449b908f14d0164fda1c in term 1.
I20260620 14:35:09.470500 17727 leader_election.cc:302] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be028979215449b908f14d0164fda1c, ab993a66f03d4d808e5e14bb9a70578e; no voters:
I20260620 14:35:09.470595 17865 raft_consensus.cc:2781] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:09.470732 17865 raft_consensus.cc:686] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 1 LEADER]: Becoming Leader. State: Replica: 8be028979215449b908f14d0164fda1c, State: Running, Role: LEADER
I20260620 14:35:09.470825 17865 consensus_queue.cc:237] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } }
I20260620 14:35:09.471621 17380 catalog_manager.cc:5495] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be028979215449b908f14d0164fda1c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8be028979215449b908f14d0164fda1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8be028979215449b908f14d0164fda1c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 } health_report { overall_health: UNKNOWN } } }
W20260620 14:35:09.490696 17573 tablet.cc:2369] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:09.511945 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.509743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.98.3:43193)
0620 14:35:09.509810 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.509812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.509820 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:35:09.509875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.509876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.509893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.509958 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:09.510075 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.510544 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.511258 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.511441 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.511489 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.511609 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:09.511707 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:09.511765 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.511795 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.511811 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:35:09.513751 17526 raft_consensus.cc:1261] T b2fd4fc739e34d1fb885e66fe7018991 P ab993a66f03d4d808e5e14bb9a70578e [term 1 FOLLOWER]: Refusing update from remote peer 8be028979215449b908f14d0164fda1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:09.513751 17660 raft_consensus.cc:1261] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 1 FOLLOWER]: Refusing update from remote peer 8be028979215449b908f14d0164fda1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:09.514155 17865 consensus_queue.cc:1035] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab993a66f03d4d808e5e14bb9a70578e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:09.514317 17867 consensus_queue.cc:1035] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86979f16c49c4e05bc536fb29cb3098a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:09.516738 17870 mvcc.cc:205] Tried to move back new op lower bound from 7298933184565473280 to 7298933184393166848. Current Snapshot: MvccSnapshot[applied={T|T < 7298933184565473280}]
I20260620 14:35:09.517267 17872 mvcc.cc:205] Tried to move back new op lower bound from 7298933184565473280 to 7298933184393166848. Current Snapshot: MvccSnapshot[applied={T|T < 7298933184565473280}]
I20260620 14:35:09.517699 17871 mvcc.cc:205] Tried to move back new op lower bound from 7298933184565473280 to 7298933184393166848. Current Snapshot: MvccSnapshot[applied={T|T < 7298933184565473280}]
I20260620 14:35:09.518836 17380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:09.518946 17380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:09.519673 17380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e246da25e07c4559a4880cde31179670: Sending DeleteTablet for 3 replicas of tablet b2fd4fc739e34d1fb885e66fe7018991
I20260620 14:35:09.520048 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39134
I20260620 14:35:09.520057 17506 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46970
I20260620 14:35:09.520066 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fd4fc739e34d1fb885e66fe7018991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51088
I20260620 14:35:09.520335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17443
I20260620 14:35:09.522224 17881 tablet_replica.cc:330] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: stopping tablet replica
I20260620 14:35:09.522351 17881 raft_consensus.cc:2229] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:09.522456 17881 raft_consensus.cc:2258] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:09.522862 17881 ts_tablet_manager.cc:1905] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:09.524288 17881 ts_tablet_manager.cc:1918] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:09.524376 17881 log.cc:1196] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/b2fd4fc739e34d1fb885e66fe7018991
I20260620 14:35:09.524623 17881 ts_tablet_manager.cc:1939] T b2fd4fc739e34d1fb885e66fe7018991 P 86979f16c49c4e05bc536fb29cb3098a: Deleting consensus metadata
I20260620 14:35:09.525535 17880 tablet_replica.cc:330] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: stopping tablet replica
I20260620 14:35:09.525645 17880 raft_consensus.cc:2229] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:35:09.525738 17880 raft_consensus.cc:2258] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:09.525887 17368 catalog_manager.cc:4841] TS 86979f16c49c4e05bc536fb29cb3098a (127.0.98.2:33711): tablet b2fd4fc739e34d1fb885e66fe7018991 (table test-table [id=232e1e0e0ead4f9c8323946e3a715175]) successfully deleted
I20260620 14:35:09.526134 17880 ts_tablet_manager.cc:1905] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:09.527447 17880 ts_tablet_manager.cc:1918] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:09.527534 17880 log.cc:1196] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/b2fd4fc739e34d1fb885e66fe7018991
I20260620 14:35:09.527818 17880 ts_tablet_manager.cc:1939] T b2fd4fc739e34d1fb885e66fe7018991 P 8be028979215449b908f14d0164fda1c: Deleting consensus metadata
W20260620 14:35:09.527802 17365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35641 (error 108)
I20260620 14:35:09.528164 17368 catalog_manager.cc:4841] TS 8be028979215449b908f14d0164fda1c (127.0.98.3:43193): tablet b2fd4fc739e34d1fb885e66fe7018991 (table test-table [id=232e1e0e0ead4f9c8323946e3a715175]) successfully deleted
I20260620 14:35:09.528285 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.528174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35641 (local address 127.0.0.1:46980)
0620 14:35:09.528220 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.528251 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:35:09.528417 17365 catalog_manager.cc:4584] TS ab993a66f03d4d808e5e14bb9a70578e (127.0.98.1:35641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2fd4fc739e34d1fb885e66fe7018991: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35641: connect: Connection refused (error 111)
I20260620 14:35:09.528623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17577
I20260620 14:35:09.534386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17711
I20260620 14:35:09.539264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17349
2026-06-20T14:35:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2040 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:09Z Disabled control of system clock
I20260620 14:35:09.566499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:09.645658 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:09.645862 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:09.645882 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:09.647055 17887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:09.647099 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:09.647111 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:09.647125 17887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:09.647172 17887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:09.648569 17887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:09.648819 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:09.649109 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:09.653877 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.653877 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.653944 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.654103 17887 server_base.cc:1034] running on GCE node
I20260620 14:35:09.654326 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:09.654564 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:09.655702 17887 hybrid_clock.cc:648] HybridClock initialized: now 1781966109655672 us; error 48 us; skew 500 ppm
I20260620 14:35:09.656905 17887 webserver.cc:460] Webserver started at http://127.0.98.62:37033/ using document root <none> and password file <none>
I20260620 14:35:09.657148 17887 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:09.657198 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:09.657374 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:09.658454 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "72b962bf6d9f42b8b3277a424493326f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f2e527d0a1267ac41bbaec5b8f417c82"
server_key_iv: "27f45c9901ffa7c246a550fdcea33ca8"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.658814 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "72b962bf6d9f42b8b3277a424493326f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f2e527d0a1267ac41bbaec5b8f417c82"
server_key_iv: "27f45c9901ffa7c246a550fdcea33ca8"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.660171 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:09.660871 17901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.661041 17887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:09.661137 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "72b962bf6d9f42b8b3277a424493326f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f2e527d0a1267ac41bbaec5b8f417c82"
server_key_iv: "27f45c9901ffa7c246a550fdcea33ca8"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.661219 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:09.671281 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:09.671567 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:09.671713 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:09.675659 17887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40033
I20260620 14:35:09.675691 17953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40033 every 8 connection(s)
I20260620 14:35:09.676015 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:09.676926 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:09.679152 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: Bootstrap starting.
I20260620 14:35:09.679664 17954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:09.679904 17954 log.cc:824] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:09.680526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17887
I20260620 14:35:09.680581 17954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: No bootstrap required, opened a new log
I20260620 14:35:09.680621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:09.680835 392 external_mini_cluster.cc:1428] Setting key d8cf0dfa8b0c50ee3190c671a56b56a8
I20260620 14:35:09.682806 17954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 14:35:09.682982 17954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:09.683020 17954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b962bf6d9f42b8b3277a424493326f, State: Initialized, Role: FOLLOWER
I20260620 14:35:09.683164 17954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 14:35:09.683251 17954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:09.683282 17954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:09.683336 17954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:09.684044 17954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 14:35:09.684149 17954 leader_election.cc:302] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b962bf6d9f42b8b3277a424493326f; no voters:
I20260620 14:35:09.684300 17954 leader_election.cc:288] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:09.684370 17959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:09.684518 17959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [term 1 LEADER]: Becoming Leader. State: Replica: 72b962bf6d9f42b8b3277a424493326f, State: Running, Role: LEADER
I20260620 14:35:09.684532 17954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:09.684602 17959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 14:35:09.684998 17960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b962bf6d9f42b8b3277a424493326f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260620 14:35:09.685081 17960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:09.685307 17961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b962bf6d9f42b8b3277a424493326f. Latest consensus state: current_term: 1 leader_uuid: "72b962bf6d9f42b8b3277a424493326f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72b962bf6d9f42b8b3277a424493326f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260620 14:35:09.685456 17961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:09.685864 17968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:09.686569 17968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:09.688079 17968 catalog_manager.cc:1347] Generated new cluster ID: 1c2270dc69194e609449ab7d5294c945
I20260620 14:35:09.688135 17968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:09.688475 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.681372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.0.98.62:40033)
0620 14:35:09.681594 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.681603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.681669 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:35:09.682762 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.682771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.682851 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.682994 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.683222 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.684960 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.686129 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.686525 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.686674 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.687654 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.687657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.687658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:09.687881 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.687920 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:09.687934 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.687995 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.688196 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.688295 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260620 14:35:09.715256 17968 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:09.715790 17968 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:09.723227 17968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: Generated new TSK 0
I20260620 14:35:09.723423 17968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:09.730400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--builtin_ntp_servers=127.0.98.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:09.820490 17978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:09.820680 17978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:09.820714 17978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:09.821889 17978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:09.821928 17978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:09.821954 17978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:09.823379 17978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:09.823647 17978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:09.823938 17978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:09.828867 17983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.828835 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.828842 17984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.829089 17978 server_base.cc:1034] running on GCE node
I20260620 14:35:09.829244 17978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:09.829493 17978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:09.830650 17978 hybrid_clock.cc:648] HybridClock initialized: now 1781966109830636 us; error 31 us; skew 500 ppm
I20260620 14:35:09.831787 17978 webserver.cc:460] Webserver started at http://127.0.98.1:34373/ using document root <none> and password file <none>
I20260620 14:35:09.832051 17978 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:09.832120 17978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:09.832271 17978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:09.833318 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "53a96f7deae643ab8c71d9a0c2eda838"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "2a7eb8f901d09d082a13611aec8d3f82"
server_key_iv: "811b092c37b601e0995c9c082f261e32"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.833732 17978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53a96f7deae643ab8c71d9a0c2eda838"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "2a7eb8f901d09d082a13611aec8d3f82"
server_key_iv: "811b092c37b601e0995c9c082f261e32"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.834992 17978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:09.835721 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.835878 17978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:09.835975 17978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "53a96f7deae643ab8c71d9a0c2eda838"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "2a7eb8f901d09d082a13611aec8d3f82"
server_key_iv: "811b092c37b601e0995c9c082f261e32"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.836053 17978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:09.844949 17978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:09.845245 17978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:09.845409 17978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:09.845643 17978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:09.845966 17978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:09.845999 17978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.846053 17978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:09.846077 17978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.851810 17978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33657
I20260620 14:35:09.851854 18105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33657 every 8 connection(s)
I20260620 14:35:09.852187 17978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:35:09.855556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 17978
I20260620 14:35:09.855666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:09.855876 392 external_mini_cluster.cc:1428] Setting key 005492d32bfab72200394b30c6a715a8
I20260620 14:35:09.856913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--builtin_ntp_servers=127.0.98.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:35:09.857693 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.852897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35775 (local address 127.0.98.62:40033)
0620 14:35:09.852977 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.852979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.853197 (+ 218us) server_negotiation.cc:407] Connection header received
0620 14:35:09.853760 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.853761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.853770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.853818 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.854447 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.854934 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.856021 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.856233 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.856289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.857324 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.857325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.857326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:09.857330 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.857360 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:09.857365 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.857460 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.857544 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.857574 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:35:09.857806 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.852867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.1:35775)
0620 14:35:09.853073 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.853106 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.853669 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.853878 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.853881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.853896 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:09.854379 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.854382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.855012 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.855013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.855918 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.855921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.856301 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.856302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.856456 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.857171 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:09.857172 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:09.857210 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:09.857241 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:09.857466 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:09.857471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:09.857474 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.857543 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.857620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260620 14:35:09.858304 18106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 14:35:09.858451 18106 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:09.858644 18106 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 14:35:09.859146 17918 ts_manager.cc:194] Registered new tserver with Master: 53a96f7deae643ab8c71d9a0c2eda838 (127.0.98.1:33657)
I20260620 14:35:09.860109 17918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35775
W20260620 14:35:09.937788 18109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:09.937986 18109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:09.938007 18109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:09.939167 18109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:09.939206 18109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:09.939231 18109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:09.940536 18109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:09.940716 18109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:09.940943 18109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:09.945793 18117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:09.945785 18114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.945851 18109 server_base.cc:1034] running on GCE node
W20260620 14:35:09.945802 18115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:09.946175 18109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:09.946416 18109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:09.947557 18109 hybrid_clock.cc:648] HybridClock initialized: now 1781966109947546 us; error 29 us; skew 500 ppm
I20260620 14:35:09.948808 18109 webserver.cc:460] Webserver started at http://127.0.98.2:33987/ using document root <none> and password file <none>
I20260620 14:35:09.949055 18109 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:09.949103 18109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:09.949251 18109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:09.950356 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "f910e6feb5bf44c19735ed565130590f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f8bc1f06c51e3b39c4c9a8f2516ac603"
server_key_iv: "e1757bfd5d54ae64c67065ed33457d84"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.950714 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f910e6feb5bf44c19735ed565130590f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f8bc1f06c51e3b39c4c9a8f2516ac603"
server_key_iv: "e1757bfd5d54ae64c67065ed33457d84"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.951972 18109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:09.952677 18123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.952893 18109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:09.952993 18109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "f910e6feb5bf44c19735ed565130590f"
format_stamp: "Formatted at 2026-06-20 14:35:09 on dist-test-slave-tc2g"
server_key: "f8bc1f06c51e3b39c4c9a8f2516ac603"
server_key_iv: "e1757bfd5d54ae64c67065ed33457d84"
server_key_version: "encryptionkey@0"
I20260620 14:35:09.953081 18109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:09.968534 18109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:09.968866 18109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:09.969022 18109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:09.969259 18109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:09.969647 18109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:09.969713 18109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.969758 18109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:09.969784 18109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:09.975421 18109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38071
I20260620 14:35:09.975467 18236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38071 every 8 connection(s)
I20260620 14:35:09.975792 18109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:35:09.980827 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.976439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37619 (local address 127.0.98.62:40033)
0620 14:35:09.976516 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:09.976519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:09.976764 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:35:09.977262 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:09.977263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:09.977274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:09.977328 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:09.978048 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.978557 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.979491 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.979676 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.979733 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.980594 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:09.980595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:09.980596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:09.980600 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:09.980614 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:09.980618 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:09.980667 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:09.980747 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:35:09.980767 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:09.981009 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:09.976420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.2:37619)
0620 14:35:09.976644 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:09.976677 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:09.977143 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:09.977410 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:09.977413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:09.977460 (+ 47us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:09.977973 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.977978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.978635 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.978635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.979415 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:09.979417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:09.979751 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:09.979752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:09.979857 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:09.980461 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:09.980462 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:09.980489 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:09.980511 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:09.980682 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:09.980686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:09.980688 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:35:09.980737 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:35:09.980817 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":123,"threads_started":1}
I20260620 14:35:09.981539 18237 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 14:35:09.981639 18237 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:09.981812 18237 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 14:35:09.982196 17918 ts_manager.cc:194] Registered new tserver with Master: f910e6feb5bf44c19735ed565130590f (127.0.98.2:38071)
I20260620 14:35:09.982424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18109
I20260620 14:35:09.982522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:09.982636 17918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37619
I20260620 14:35:09.982738 392 external_mini_cluster.cc:1428] Setting key d296352cef341113eee382d87b40ec29
I20260620 14:35:09.983566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--builtin_ntp_servers=127.0.98.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:10.063570 18240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:10.063766 18240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:10.063787 18240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:10.064955 18240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:10.065030 18240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:10.065058 18240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:10.066382 18240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:10.066569 18240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:10.066800 18240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:10.071707 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:10.071722 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:10.071810 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:10.071831 18240 server_base.cc:1034] running on GCE node
I20260620 14:35:10.072201 18240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:10.072427 18240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:10.073585 18240 hybrid_clock.cc:648] HybridClock initialized: now 1781966110073575 us; error 28 us; skew 500 ppm
I20260620 14:35:10.074844 18240 webserver.cc:460] Webserver started at http://127.0.98.3:41501/ using document root <none> and password file <none>
I20260620 14:35:10.075088 18240 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:10.075160 18240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:10.075295 18240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:10.076346 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "a75cf2911fd6480bbbed2e8101672f0b"
format_stamp: "Formatted at 2026-06-20 14:35:10 on dist-test-slave-tc2g"
server_key: "5283812d13203bab1cd0ea3fce6369cd"
server_key_iv: "1fe9fb3cd651dce48228ee871ac53413"
server_key_version: "encryptionkey@0"
I20260620 14:35:10.076702 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a75cf2911fd6480bbbed2e8101672f0b"
format_stamp: "Formatted at 2026-06-20 14:35:10 on dist-test-slave-tc2g"
server_key: "5283812d13203bab1cd0ea3fce6369cd"
server_key_iv: "1fe9fb3cd651dce48228ee871ac53413"
server_key_version: "encryptionkey@0"
I20260620 14:35:10.078068 18240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:10.078747 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:10.078934 18240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:10.079030 18240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "a75cf2911fd6480bbbed2e8101672f0b"
format_stamp: "Formatted at 2026-06-20 14:35:10 on dist-test-slave-tc2g"
server_key: "5283812d13203bab1cd0ea3fce6369cd"
server_key_iv: "1fe9fb3cd651dce48228ee871ac53413"
server_key_version: "encryptionkey@0"
I20260620 14:35:10.079111 18240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:10.097900 18240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:10.098276 18240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:10.098459 18240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:10.098701 18240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:10.099053 18240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:10.099138 18240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:10.099179 18240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:10.099207 18240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:10.105028 18240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40299
I20260620 14:35:10.105070 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40299 every 8 connection(s)
I20260620 14:35:10.105409 18240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:10.108388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18240
I20260620 14:35:10.108464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:35:10.108664 392 external_mini_cluster.cc:1428] Setting key 78a9ab07390a118136fac015e44943e7
I20260620 14:35:10.111099 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59065 (local address 127.0.98.62:40033)
0620 14:35:10.106220 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.106222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.106457 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:35:10.107007 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.107008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.107020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.107076 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:10.107766 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.108299 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.109712 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.109935 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.110012 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.110833 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:10.110835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:10.110835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:10.110839 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:10.110856 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:10.110860 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:10.110909 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.110991 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.111019 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:35:10.111321 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.106124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.3:59065)
0620 14:35:10.106337 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.106370 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.106893 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.107146 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.107150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.107160 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:10.107661 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.107673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.108397 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.108398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.109607 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.109609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.110017 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.110018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.110171 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.110659 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:10.110659 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:10.110691 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:10.110723 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:10.110915 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:10.110919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:10.110922 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.110985 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.111090 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260620 14:35:10.111766 18368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 14:35:10.111843 18368 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:10.111984 18368 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 14:35:10.112296 17918 ts_manager.cc:194] Registered new tserver with Master: a75cf2911fd6480bbbed2e8101672f0b (127.0.98.3:40299)
I20260620 14:35:10.112675 17918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59065
I20260620 14:35:10.120519 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:10.124800 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.121888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.62:40033)
0620 14:35:10.121948 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.121950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.122084 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:35:10.122123 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.122124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.122132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.122171 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:10.122326 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.122773 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.123611 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.123783 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.123837 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.124526 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:10.124527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:10.124527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:10.124531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:10.124545 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:10.124548 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:10.124600 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.124650 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.124669 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:35:10.126375 17918 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:10.127871 17918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:10.134325 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.129999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33657 (local address 127.0.0.1:53450)
0620 14:35:10.130443 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.130452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.130511 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.130694 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.130697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.130699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:10.130763 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.130764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.132024 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.132024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.133085 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.133087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.134040 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.134041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.134117 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.134131 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.134205 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.134249 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":28,"thread_start_us":54,"threads_started":1}
I20260620 14:35:10.134354 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.130159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.98.1:33657)
0620 14:35:10.130385 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.130390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.130524 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:35:10.130580 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.130582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.130595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.130683 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:10.130830 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.131950 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.133185 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.133950 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.134029 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.134122 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.134218 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.134256 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260620 14:35:10.134667 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.130166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.98.3:40299)
0620 14:35:10.130637 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.130641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.130651 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:10.130686 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.130688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.130704 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.130933 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:10.130984 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.132020 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.133351 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.134054 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.134111 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.134182 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.134578 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.134611 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":119,"threads_started":1}
I20260620 14:35:10.134857 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.129999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40299 (local address 127.0.0.1:46800)
0620 14:35:10.130271 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.130312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.130389 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.130803 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.130804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.130806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:10.130859 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.130861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.132079 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.132080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.133263 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.133266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.134216 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.134217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.134284 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.134287 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.134339 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.134381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260620 14:35:10.135135 18040 tablet_service.cc:1467] Processing CreateTablet for tablet 3741d4564aa048c8b2df2a73f955e46d (DEFAULT_TABLE table=test-table [id=668f706e841d4267b6fa484ede361e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:10.135536 18040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3741d4564aa048c8b2df2a73f955e46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:10.135529 18302 tablet_service.cc:1467] Processing CreateTablet for tablet 3741d4564aa048c8b2df2a73f955e46d (DEFAULT_TABLE table=test-table [id=668f706e841d4267b6fa484ede361e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:10.135874 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.130007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38071 (local address 127.0.0.1:55466)
0620 14:35:10.130495 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.130504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.130540 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.132337 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.132337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.132340 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:10.132373 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.132374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.133586 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.133586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.134596 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.134599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.135517 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.135519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.135650 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.135655 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.135769 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.135807 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":107,"threads_started":1}
I20260620 14:35:10.135902 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3741d4564aa048c8b2df2a73f955e46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:10.136037 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.130267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.98.2:38071)
0620 14:35:10.132162 (+ 1895us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.132166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.132175 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:10.132210 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.132212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.132227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.132324 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:10.132437 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.133519 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.134682 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.135426 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.135503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.135888 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.135915 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.135956 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1858,"thread_start_us":67,"threads_started":1}
I20260620 14:35:10.136615 18171 tablet_service.cc:1467] Processing CreateTablet for tablet 3741d4564aa048c8b2df2a73f955e46d (DEFAULT_TABLE table=test-table [id=668f706e841d4267b6fa484ede361e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:10.136943 18171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3741d4564aa048c8b2df2a73f955e46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:10.139195 18383 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Bootstrap starting.
I20260620 14:35:10.139763 18383 tablet_bootstrap.cc:654] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:10.140031 18383 log.cc:824] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:10.140054 18384 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Bootstrap starting.
I20260620 14:35:10.140494 18382 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: Bootstrap starting.
I20260620 14:35:10.140717 18384 tablet_bootstrap.cc:654] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:10.140775 18383 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: No bootstrap required, opened a new log
I20260620 14:35:10.140839 18383 ts_tablet_manager.cc:1397] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:10.141041 18384 log.cc:824] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:10.141111 18382 tablet_bootstrap.cc:654] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:10.141422 18382 log.cc:824] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:10.141804 18384 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: No bootstrap required, opened a new log
I20260620 14:35:10.141889 18384 ts_tablet_manager.cc:1397] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:10.142256 18382 tablet_bootstrap.cc:492] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: No bootstrap required, opened a new log
I20260620 14:35:10.142377 18382 ts_tablet_manager.cc:1397] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:10.143105 18383 raft_consensus.cc:348] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.143239 18383 raft_consensus.cc:374] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:10.143261 18383 raft_consensus.cc:729] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75cf2911fd6480bbbed2e8101672f0b, State: Initialized, Role: FOLLOWER
I20260620 14:35:10.143374 18383 consensus_queue.cc:260] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.143700 18368 heartbeater.cc:503] Master 127.0.98.62:40033 was elected leader, sending a full tablet report...
I20260620 14:35:10.144002 18383 ts_tablet_manager.cc:1428] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:35:10.143947 18384 raft_consensus.cc:348] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.144095 18384 raft_consensus.cc:374] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:10.144129 18384 raft_consensus.cc:729] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f910e6feb5bf44c19735ed565130590f, State: Initialized, Role: FOLLOWER
I20260620 14:35:10.144230 18384 consensus_queue.cc:260] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.144474 18384 ts_tablet_manager.cc:1428] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:10.144587 18237 heartbeater.cc:503] Master 127.0.98.62:40033 was elected leader, sending a full tablet report...
I20260620 14:35:10.145784 18382 raft_consensus.cc:348] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.145905 18382 raft_consensus.cc:374] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:10.145926 18382 raft_consensus.cc:729] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a96f7deae643ab8c71d9a0c2eda838, State: Initialized, Role: FOLLOWER
I20260620 14:35:10.146062 18382 consensus_queue.cc:260] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.146301 18382 ts_tablet_manager.cc:1428] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:35:10.146358 18106 heartbeater.cc:503] Master 127.0.98.62:40033 was elected leader, sending a full tablet report...
I20260620 14:35:10.155328 18390 raft_consensus.cc:482] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:10.155402 18390 raft_consensus.cc:504] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.155718 18390 leader_election.cc:288] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a75cf2911fd6480bbbed2e8101672f0b (127.0.98.3:40299), f910e6feb5bf44c19735ed565130590f (127.0.98.2:38071)
I20260620 14:35:10.158568 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.155918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53743 (local address 127.0.98.2:38071)
0620 14:35:10.155958 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.155960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.155969 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:10.156005 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.156006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.156014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.156093 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:10.156232 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.156807 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.157837 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.158239 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.158291 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.158299 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.158453 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.158506 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:35:10.158566 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.155857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38071 (local address 127.0.98.1:53743)
0620 14:35:10.155909 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.155921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.155958 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.156097 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.156097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.156099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:10.156161 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.156163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.156880 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.156882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.157761 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.157764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.158310 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.158311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.158379 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.158382 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.158444 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.158492 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:35:10.158814 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.155918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38293 (local address 127.0.98.3:40299)
0620 14:35:10.155969 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.155971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.156085 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:35:10.156166 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.156168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.156178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.156258 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:10.156432 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.156989 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.158013 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.158432 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.158491 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.158499 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.158720 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.158760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:35:10.158815 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.155868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40299 (local address 127.0.98.1:38293)
0620 14:35:10.156029 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.156039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:10.156107 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:10.156268 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:10.156269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:10.156271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:10.156336 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.156337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.157066 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.157067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.157933 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:10.157935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.158504 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.158505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:10.158613 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.158617 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:10.158708 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:35:10.158743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260620 14:35:10.158893 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3741d4564aa048c8b2df2a73f955e46d" candidate_uuid: "53a96f7deae643ab8c71d9a0c2eda838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f910e6feb5bf44c19735ed565130590f" is_pre_election: true
I20260620 14:35:10.159036 18191 raft_consensus.cc:2454] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a96f7deae643ab8c71d9a0c2eda838 in term 0.
I20260620 14:35:10.159094 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3741d4564aa048c8b2df2a73f955e46d" candidate_uuid: "53a96f7deae643ab8c71d9a0c2eda838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75cf2911fd6480bbbed2e8101672f0b" is_pre_election: true
I20260620 14:35:10.159246 18322 raft_consensus.cc:2454] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a96f7deae643ab8c71d9a0c2eda838 in term 0.
I20260620 14:35:10.159288 17993 leader_election.cc:302] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a96f7deae643ab8c71d9a0c2eda838, f910e6feb5bf44c19735ed565130590f; no voters:
I20260620 14:35:10.159510 18390 raft_consensus.cc:2781] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:10.159581 18390 raft_consensus.cc:482] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:10.159605 18390 raft_consensus.cc:3037] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:10.160218 18390 raft_consensus.cc:504] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.160360 18390 leader_election.cc:288] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [CANDIDATE]: Term 1 election: Requested vote from peers a75cf2911fd6480bbbed2e8101672f0b (127.0.98.3:40299), f910e6feb5bf44c19735ed565130590f (127.0.98.2:38071)
I20260620 14:35:10.160532 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3741d4564aa048c8b2df2a73f955e46d" candidate_uuid: "53a96f7deae643ab8c71d9a0c2eda838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75cf2911fd6480bbbed2e8101672f0b"
I20260620 14:35:10.160537 18191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3741d4564aa048c8b2df2a73f955e46d" candidate_uuid: "53a96f7deae643ab8c71d9a0c2eda838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f910e6feb5bf44c19735ed565130590f"
I20260620 14:35:10.160607 18322 raft_consensus.cc:3037] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:10.160626 18191 raft_consensus.cc:3037] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:10.161280 18322 raft_consensus.cc:2454] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a96f7deae643ab8c71d9a0c2eda838 in term 1.
I20260620 14:35:10.161440 18191 raft_consensus.cc:2454] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a96f7deae643ab8c71d9a0c2eda838 in term 1.
I20260620 14:35:10.161501 17995 leader_election.cc:302] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a96f7deae643ab8c71d9a0c2eda838, a75cf2911fd6480bbbed2e8101672f0b; no voters:
I20260620 14:35:10.161587 18390 raft_consensus.cc:2781] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:10.161723 18390 raft_consensus.cc:686] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [term 1 LEADER]: Becoming Leader. State: Replica: 53a96f7deae643ab8c71d9a0c2eda838, State: Running, Role: LEADER
I20260620 14:35:10.161819 18390 consensus_queue.cc:237] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } }
I20260620 14:35:10.162559 17918 catalog_manager.cc:5495] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53a96f7deae643ab8c71d9a0c2eda838 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "53a96f7deae643ab8c71d9a0c2eda838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a96f7deae643ab8c71d9a0c2eda838" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 } health_report { overall_health: UNKNOWN } } }
I20260620 14:35:10.170861 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.168832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.1:33657)
0620 14:35:10.168884 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:10.168886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:10.168892 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:35:10.168916 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:10.168917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:10.168931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:10.168988 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:10.169092 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.169587 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.170245 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:10.170417 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:10.170462 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:10.170582 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:10.170684 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:10.170742 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:10.170779 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:35:10.170795 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:35:10.172384 18191 raft_consensus.cc:1261] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 1 FOLLOWER]: Refusing update from remote peer 53a96f7deae643ab8c71d9a0c2eda838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:10.172396 18322 raft_consensus.cc:1261] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 1 FOLLOWER]: Refusing update from remote peer 53a96f7deae643ab8c71d9a0c2eda838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:10.172693 18392 consensus_queue.cc:1035] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f910e6feb5bf44c19735ed565130590f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:10.172819 18390 consensus_queue.cc:1035] T 3741d4564aa048c8b2df2a73f955e46d P 53a96f7deae643ab8c71d9a0c2eda838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a75cf2911fd6480bbbed2e8101672f0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:10.176169 18397 mvcc.cc:205] Tried to move back new op lower bound from 7298933187263926272 to 7298933187223527424. Current Snapshot: MvccSnapshot[applied={T|T < 7298933187263926272 or (T in {7298933187263926272})}]
I20260620 14:35:10.177126 17918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:10.177199 17918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:10.177939 17918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72b962bf6d9f42b8b3277a424493326f: Sending DeleteTablet for 3 replicas of tablet 3741d4564aa048c8b2df2a73f955e46d
I20260620 14:35:10.178237 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet 3741d4564aa048c8b2df2a73f955e46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:10 UTC) from {username='slave'} at 127.0.0.1:46800
I20260620 14:35:10.178236 18040 tablet_service.cc:1514] Processing DeleteTablet for tablet 3741d4564aa048c8b2df2a73f955e46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:10 UTC) from {username='slave'} at 127.0.0.1:53450
I20260620 14:35:10.178532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17978
I20260620 14:35:10.178573 18171 tablet_service.cc:1514] Processing DeleteTablet for tablet 3741d4564aa048c8b2df2a73f955e46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:10 UTC) from {username='slave'} at 127.0.0.1:55466
I20260620 14:35:10.179745 18406 tablet_replica.cc:330] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: stopping tablet replica
I20260620 14:35:10.179857 18406 raft_consensus.cc:2229] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:10.179956 18406 raft_consensus.cc:2258] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:10.180327 18406 ts_tablet_manager.cc:1905] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:10.181696 18406 ts_tablet_manager.cc:1918] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:10.181783 18406 log.cc:1196] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/3741d4564aa048c8b2df2a73f955e46d
I20260620 14:35:10.182008 18406 ts_tablet_manager.cc:1939] T 3741d4564aa048c8b2df2a73f955e46d P f910e6feb5bf44c19735ed565130590f: Deleting consensus metadata
I20260620 14:35:10.182452 17902 catalog_manager.cc:4841] TS f910e6feb5bf44c19735ed565130590f (127.0.98.2:38071): tablet 3741d4564aa048c8b2df2a73f955e46d (table test-table [id=668f706e841d4267b6fa484ede361e89]) successfully deleted
W20260620 14:35:10.183851 17905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33657 (error 108)
I20260620 14:35:10.184022 18404 tablet_replica.cc:330] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: stopping tablet replica
I20260620 14:35:10.184137 18404 raft_consensus.cc:2229] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:10.184233 18404 raft_consensus.cc:2258] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:10.184289 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:10.184172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33657 (local address 127.0.0.1:53458)
0620 14:35:10.184225 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:10.184266 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:35:10.184408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18109
W20260620 14:35:10.184494 17905 catalog_manager.cc:4584] TS 53a96f7deae643ab8c71d9a0c2eda838 (127.0.98.1:33657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3741d4564aa048c8b2df2a73f955e46d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33657: connect: Connection refused (error 111)
I20260620 14:35:10.185289 18404 ts_tablet_manager.cc:1905] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:10.187240 18404 ts_tablet_manager.cc:1918] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:10.187320 18404 log.cc:1196] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/3741d4564aa048c8b2df2a73f955e46d
I20260620 14:35:10.187564 18404 ts_tablet_manager.cc:1939] T 3741d4564aa048c8b2df2a73f955e46d P a75cf2911fd6480bbbed2e8101672f0b: Deleting consensus metadata
I20260620 14:35:10.187955 17904 catalog_manager.cc:4841] TS a75cf2911fd6480bbbed2e8101672f0b (127.0.98.3:40299): tablet 3741d4564aa048c8b2df2a73f955e46d (table test-table [id=668f706e841d4267b6fa484ede361e89]) successfully deleted
I20260620 14:35:10.190135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18240
I20260620 14:35:10.195029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 17887
2026-06-20T14:35:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (654 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:10 dist-test-slave-tc2g krb5kdc[18410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:10 dist-test-slave-tc2g krb5kdc[18410](info): set up 2 sockets
Jun 20 14:35:10 dist-test-slave-tc2g krb5kdc[18410](info): commencing operation
krb5kdc: starting...
W20260620 14:35:11.249284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:11.361793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:11.441133 18423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:11.441370 18423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:11.441407 18423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:11.442539 18423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:11.442606 18423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:11.442632 18423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:11.442665 18423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:11.442690 18423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:11.444126 18423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:11.444396 18423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:11.444607 18423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:11.449357 18429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.449401 18428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.449587 18423 server_base.cc:1034] running on GCE node
W20260620 14:35:11.449685 18431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.449914 18423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:11.450196 18423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:11.451339 18423 hybrid_clock.cc:648] HybridClock initialized: now 1781966111451330 us; error 29 us; skew 500 ppm
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:11.453712 18423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:11.454167 18423 webserver.cc:460] Webserver started at http://127.0.98.62:44387/ using document root <none> and password file <none>
I20260620 14:35:11.454411 18423 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:11.454479 18423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:11.454627 18423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:11.455667 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "626095b0c34d486ca84d45634ab01007"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "1c2e9605f62d019dbb39bd45cc13d209"
server_key_iv: "6c5b66b40e8b916f4de380f4d73e570b"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.456027 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "626095b0c34d486ca84d45634ab01007"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "1c2e9605f62d019dbb39bd45cc13d209"
server_key_iv: "6c5b66b40e8b916f4de380f4d73e570b"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.457330 18423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:11.458068 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.458241 18423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:11.458355 18423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "626095b0c34d486ca84d45634ab01007"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "1c2e9605f62d019dbb39bd45cc13d209"
server_key_iv: "6c5b66b40e8b916f4de380f4d73e570b"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.458436 18423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:11.466117 18423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:11.467844 18423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:11.468006 18423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:11.471946 18423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260620 14:35:11.471982 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260620 14:35:11.472342 18423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:11.473254 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:11.475497 18491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: Bootstrap starting.
I20260620 14:35:11.475775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18423
I20260620 14:35:11.475912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:11.476099 392 external_mini_cluster.cc:1428] Setting key 3604bc2fdc072bb79113976fe639f823
I20260620 14:35:11.476734 18491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:11.477083 18491 log.cc:824] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:11.477996 18491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: No bootstrap required, opened a new log
I20260620 14:35:11.480654 18491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 14:35:11.480887 18491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:11.480966 18491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626095b0c34d486ca84d45634ab01007, State: Initialized, Role: FOLLOWER
I20260620 14:35:11.481132 18491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 14:35:11.481222 18491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:11.481261 18491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:11.481310 18491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:11.482064 18491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 14:35:11.482170 18491 leader_election.cc:302] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626095b0c34d486ca84d45634ab01007; no voters:
I20260620 14:35:11.482326 18491 leader_election.cc:288] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:11.482460 18496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:11.482601 18491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:11.482771 18496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [term 1 LEADER]: Becoming Leader. State: Replica: 626095b0c34d486ca84d45634ab01007, State: Running, Role: LEADER
I20260620 14:35:11.482982 18496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 14:35:11.483384 18496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626095b0c34d486ca84d45634ab01007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 14:35:11.483481 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626095b0c34d486ca84d45634ab01007. Latest consensus state: current_term: 1 leader_uuid: "626095b0c34d486ca84d45634ab01007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626095b0c34d486ca84d45634ab01007" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 14:35:11.483549 18496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:11.483563 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:11.483870 18505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:11.484369 18505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:11.486047 18505 catalog_manager.cc:1347] Generated new cluster ID: 440c46958eaf4da7ab09e3d14561ecad
I20260620 14:35:11.486099 18505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:11.489322 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.476987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.98.62:43019)
0620 14:35:11.477460 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.477470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.477502 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:35:11.478149 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.478157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.478217 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.478362 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:11.479326 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.480672 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.482441 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.483076 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.483174 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.486110 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:11.486114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:11.486115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:11.486127 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:11.487837 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.488235 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.488237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.488238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.488323 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.488537 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.488538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.488538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.488765 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:11.488945 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.489022 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.489156 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":69,"threads_started":1}
I20260620 14:35:11.492957 18505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:11.493669 18505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:11.504082 18505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: Generated new TSK 0
I20260620 14:35:11.504281 18505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:35:11.532418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:11.614123 18517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:11.614318 18517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:11.614351 18517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:11.615623 18517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:11.615667 18517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:11.615717 18517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:11.617282 18517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:11.617548 18517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:11.617780 18517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:11.622586 18525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.622632 18523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.622692 18517 server_base.cc:1034] running on GCE node
W20260620 14:35:11.622788 18522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.622984 18517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:11.623226 18517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:11.624387 18517 hybrid_clock.cc:648] HybridClock initialized: now 1781966111624385 us; error 36 us; skew 500 ppm
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:11.626904 18517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:35:11.627377 18517 webserver.cc:460] Webserver started at http://127.0.98.1:43419/ using document root <none> and password file <none>
I20260620 14:35:11.627610 18517 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:11.627655 18517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:11.627807 18517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:11.628844 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "54371f65c2a24222bc577509c08b3a1c"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "9c31fbeb22fb145c593083db344d3591"
server_key_iv: "e9926cfa116553d77a143fdc47a299a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.629212 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54371f65c2a24222bc577509c08b3a1c"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "9c31fbeb22fb145c593083db344d3591"
server_key_iv: "e9926cfa116553d77a143fdc47a299a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.630573 18517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:11.631282 18532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.631434 18517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:11.631517 18517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "54371f65c2a24222bc577509c08b3a1c"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "9c31fbeb22fb145c593083db344d3591"
server_key_iv: "e9926cfa116553d77a143fdc47a299a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.631569 18517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:11.639185 18517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:11.640895 18517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:11.641003 18517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:11.641178 18517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:11.641414 18517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:11.641507 18517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.641531 18517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:11.641552 18517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.647401 18517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41451
I20260620 14:35:11.647475 18645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41451 every 8 connection(s)
I20260620 14:35:11.647795 18517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:11.656883 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.648510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49945 (local address 127.0.98.62:43019)
0620 14:35:11.648575 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.648591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.648768 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:35:11.649825 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.649826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.649839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.649887 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:11.650517 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.651022 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.652034 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.652260 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.652343 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.654389 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:11.654391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:11.654391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:11.654398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:11.655777 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.656118 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.656119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.656119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.656168 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.656366 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.656367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.656367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.656469 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:11.656519 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.656746 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.656810 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:35:11.657021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18517
I20260620 14:35:11.657127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:11.657121 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.648490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:49945)
0620 14:35:11.648647 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.648691 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.649710 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.649967 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.649971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.650216 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:11.650408 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.650412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.651115 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.651116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.651930 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.651932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.652354 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.652355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.652478 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.652931 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:11.652939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:11.654297 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:11.655849 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.655854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.655856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.656055 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.656224 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.656225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.656225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.656315 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.656523 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:11.656527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:11.656639 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.656719 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.656883 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260620 14:35:11.657353 392 external_mini_cluster.cc:1428] Setting key b61bd1c108d13e76731aa9f11e671fbb
I20260620 14:35:11.657642 18646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 14:35:11.657766 18646 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:11.657949 18646 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 14:35:11.658505 18455 ts_manager.cc:194] Registered new tserver with Master: 54371f65c2a24222bc577509c08b3a1c (127.0.98.1:41451)
I20260620 14:35:11.659107 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49945
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:35:11.680485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:11.760846 18651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:11.761052 18651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:11.761075 18651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:11.762315 18651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:11.762356 18651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:11.762385 18651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:11.763797 18651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:11.763995 18651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:11.764183 18651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:11.769016 18656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.769022 18657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.769021 18659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.769313 18651 server_base.cc:1034] running on GCE node
I20260620 14:35:11.769577 18651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:11.769809 18651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:11.770964 18651 hybrid_clock.cc:648] HybridClock initialized: now 1781966111770953 us; error 26 us; skew 500 ppm
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:11.773365 18651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:35:11.773854 18651 webserver.cc:460] Webserver started at http://127.0.98.2:37137/ using document root <none> and password file <none>
I20260620 14:35:11.774078 18651 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:11.774147 18651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:11.774281 18651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:11.775285 18651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b5f83e50e4e4049a99c1ded451e3947"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "21e35ea3cf26a5c91f193ab814085d08"
server_key_iv: "7bf8cd1cd514f0f1db910fc827480231"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.775655 18651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b5f83e50e4e4049a99c1ded451e3947"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "21e35ea3cf26a5c91f193ab814085d08"
server_key_iv: "7bf8cd1cd514f0f1db910fc827480231"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.776954 18651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:11.777734 18666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.777956 18651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:11.778053 18651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "1b5f83e50e4e4049a99c1ded451e3947"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "21e35ea3cf26a5c91f193ab814085d08"
server_key_iv: "7bf8cd1cd514f0f1db910fc827480231"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.778134 18651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:11.789860 18651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:11.791632 18651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:11.791790 18651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:11.792012 18651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:11.792306 18651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:11.792361 18651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.792416 18651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:11.792445 18651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.798457 18651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42579
I20260620 14:35:11.798511 18779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42579 every 8 connection(s)
I20260620 14:35:11.798854 18651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:11.805214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18651
I20260620 14:35:11.805335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:11.805594 392 external_mini_cluster.cc:1428] Setting key 0bc97489e50c8fe3353310923e227722
I20260620 14:35:11.808485 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.799539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46325 (local address 127.0.98.62:43019)
0620 14:35:11.799604 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.799606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.799815 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:35:11.800758 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.800759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.800772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.800831 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:11.801455 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.801957 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.802970 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.803167 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.803250 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.805405 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:11.805407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:11.805407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:11.805415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:11.807141 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.807522 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.807523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.807524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.807586 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.807885 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.807886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.807887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.807988 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:11.808043 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.808360 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.808424 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:35:11.808751 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.799524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:46325)
0620 14:35:11.799693 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.799728 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.800642 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.800908 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.800911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.801158 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:11.801338 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.801342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.802047 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.802048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.802864 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.802866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.803258 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.803260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.803380 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.803826 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:11.803834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:11.805259 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:11.807223 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.807228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.807230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.807446 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.807663 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.807664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.807664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.807811 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.808053 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:11.808059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:11.808220 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.808337 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.808504 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260620 14:35:11.809242 18780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 14:35:11.809347 18780 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:11.809544 18780 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 14:35:11.809911 18455 ts_manager.cc:194] Registered new tserver with Master: 1b5f83e50e4e4049a99c1ded451e3947 (127.0.98.2:42579)
I20260620 14:35:11.810448 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46325
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:35:11.830101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:11.909890 18785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:11.910133 18785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:11.910171 18785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:11.911294 18785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:11.911361 18785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:11.911406 18785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:11.912897 18785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:11.913172 18785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:11.913399 18785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:11.918166 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.918125 18790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:11.918133 18791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:11.918401 18785 server_base.cc:1034] running on GCE node
I20260620 14:35:11.918638 18785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:11.918869 18785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:11.920032 18785 hybrid_clock.cc:648] HybridClock initialized: now 1781966111920024 us; error 38 us; skew 500 ppm
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:11.922434 18785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:35:11.922906 18785 webserver.cc:460] Webserver started at http://127.0.98.3:33997/ using document root <none> and password file <none>
I20260620 14:35:11.923130 18785 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:11.923197 18785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:11.923349 18785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:11.924342 18785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e3ff928ede147e68ca0f173e0bd323f"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "0a784ce1926dc051c96c7362f96bc31d"
server_key_iv: "262ac4f562f06e39f6fe0fd4926aff03"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.924700 18785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e3ff928ede147e68ca0f173e0bd323f"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "0a784ce1926dc051c96c7362f96bc31d"
server_key_iv: "262ac4f562f06e39f6fe0fd4926aff03"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.926072 18785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:11.926757 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.926934 18785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:11.927033 18785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "2e3ff928ede147e68ca0f173e0bd323f"
format_stamp: "Formatted at 2026-06-20 14:35:11 on dist-test-slave-tc2g"
server_key: "0a784ce1926dc051c96c7362f96bc31d"
server_key_iv: "262ac4f562f06e39f6fe0fd4926aff03"
server_key_version: "encryptionkey@0"
I20260620 14:35:11.927109 18785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:11.941696 18785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:11.943379 18785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:11.943486 18785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:11.943643 18785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:11.943957 18785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:11.944018 18785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.944054 18785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:11.944103 18785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:11.949852 18785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38623
I20260620 14:35:11.949898 18913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38623 every 8 connection(s)
I20260620 14:35:11.950199 18785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:11.954584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18785
I20260620 14:35:11.954707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:35:11.955267 392 external_mini_cluster.cc:1428] Setting key 205266cbb847ea7be3465948d341e937
Jun 20 14:35:11 dist-test-slave-tc2g krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:11.959137 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.950876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49085 (local address 127.0.98.62:43019)
0620 14:35:11.950956 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.950959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.951137 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:35:11.952024 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.952025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.952035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.952088 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:11.952672 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.953172 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.954132 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.954328 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.954381 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.956822 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:11.956823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:11.956823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:11.956830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:11.958028 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.958403 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.958404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.958404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.958453 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.958653 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.958653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.958654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.958749 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:11.958798 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.959017 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.959074 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:35:11.959317 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.950833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:49085)
0620 14:35:11.951014 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.951048 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.951935 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.952153 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.952156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.952406 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:11.952599 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.952603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.953248 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.953248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.954058 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.954060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.954423 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.954424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.954579 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.955141 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:11.955153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:11.956743 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:11.958104 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.958110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.958112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.958329 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.958506 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:11.958506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:11.958507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:11.958601 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:11.958802 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:11.958806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:11.958918 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.958999 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.959113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260620 14:35:11.959806 18914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 14:35:11.959913 18914 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:11.960084 18914 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 14:35:11.960507 18455 ts_manager.cc:194] Registered new tserver with Master: 2e3ff928ede147e68ca0f173e0bd323f (127.0.98.3:38623)
I20260620 14:35:11.960891 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49085
I20260620 14:35:11.967075 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:11.975415 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.968444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.0.98.62:43019)
0620 14:35:11.968507 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.968509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.968618 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:35:11.968655 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.968656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.968663 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.968713 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:11.969325 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.969839 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.970543 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.970714 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.970761 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.972386 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:11.972388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:11.972389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:11.972397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:11.974156 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.974597 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.974599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.974599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.974660 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:11.974932 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:11.974933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:11.974934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:11.975114 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:11.975194 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.975305 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.975354 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:35:11.977046 18455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:11.978655 18455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:11.984738 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.980602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42579 (local address 127.0.0.1:49164)
0620 14:35:11.981101 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.981112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.981171 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.981451 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.981454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.981456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:11.981534 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.981536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.982531 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.982532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.983674 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.983676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.984534 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.984535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.984599 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.984610 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.984646 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.984682 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":59,"threads_started":1}
I20260620 14:35:11.985044 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.980957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.98.2:42579)
0620 14:35:11.981111 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.981116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.981162 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:35:11.981245 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.981247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.981266 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.981614 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:11.981718 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.982424 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.983754 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.984403 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.984866 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.984938 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.984959 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.984991 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260620 14:35:11.985531 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.980582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41451 (local address 127.0.0.1:38294)
0620 14:35:11.980703 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.980748 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.980807 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.981536 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.981537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.981539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:11.981585 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.981587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.982954 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.982956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.984130 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.984132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.985299 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.985300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.985365 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.985370 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.985400 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.985461 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260620 14:35:11.985720 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.980804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38623 (local address 127.0.0.1:53936)
0620 14:35:11.981347 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:11.981357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:11.981406 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:11.981730 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:11.981731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:11.981732 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:11.981768 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.981769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.983062 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.983062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.984186 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:11.984187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.985495 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.985497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:11.985557 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.985562 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:35:11.985617 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:35:11.985654 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":69,"threads_started":1}
I20260620 14:35:11.985755 18714 tablet_service.cc:1467] Processing CreateTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd (DEFAULT_TABLE table=test-table [id=ec3bd36054924830b13744173d478ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:11.985906 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.981169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.0.98.1:41451)
0620 14:35:11.981345 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.981349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.981360 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:11.981399 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.981400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.981415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.981660 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:11.981756 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.982831 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.984260 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.985199 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.985672 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.985737 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.985763 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.985816 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260620 14:35:11.985980 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:11.980914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.0.98.3:38623)
0620 14:35:11.981086 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:11.981090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:11.981473 (+ 383us) server_negotiation.cc:407] Connection header received
0620 14:35:11.981563 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:11.981565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:11.981582 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:11.981650 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:11.981844 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.982944 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.984269 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:11.985300 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:11.985357 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:11.985460 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:11.985863 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 14:35:11.985904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 14:35:11.986124 18714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b9dadf42a9470faa22a0d4f1b674cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:11.986624 18848 tablet_service.cc:1467] Processing CreateTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd (DEFAULT_TABLE table=test-table [id=ec3bd36054924830b13744173d478ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:11.986934 18848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b9dadf42a9470faa22a0d4f1b674cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:11.989645 18928 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Bootstrap starting.
I20260620 14:35:11.990198 18580 tablet_service.cc:1467] Processing CreateTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd (DEFAULT_TABLE table=test-table [id=ec3bd36054924830b13744173d478ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:11.990383 18928 tablet_bootstrap.cc:654] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:11.990551 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b9dadf42a9470faa22a0d4f1b674cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:11.990928 18928 log.cc:824] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:11.991273 18929 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Bootstrap starting.
I20260620 14:35:11.991701 18928 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: No bootstrap required, opened a new log
I20260620 14:35:11.991777 18928 ts_tablet_manager.cc:1397] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:11.993945 18931 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Bootstrap starting.
I20260620 14:35:11.993992 18929 tablet_bootstrap.cc:654] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:11.993964 18928 raft_consensus.cc:348] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.994170 18928 raft_consensus.cc:374] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:11.994221 18928 raft_consensus.cc:729] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5f83e50e4e4049a99c1ded451e3947, State: Initialized, Role: FOLLOWER
I20260620 14:35:11.994349 18928 consensus_queue.cc:260] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.994632 18928 ts_tablet_manager.cc:1428] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:11.994637 18931 tablet_bootstrap.cc:654] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:11.994755 18780 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 14:35:11.994813 18929 log.cc:824] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:11.995185 18931 log.cc:824] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:11.995630 18929 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: No bootstrap required, opened a new log
I20260620 14:35:11.995713 18929 ts_tablet_manager.cc:1397] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:35:11.996012 18931 tablet_bootstrap.cc:492] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: No bootstrap required, opened a new log
I20260620 14:35:11.996076 18931 ts_tablet_manager.cc:1397] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:11.997676 18929 raft_consensus.cc:348] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.997865 18929 raft_consensus.cc:374] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:11.997900 18929 raft_consensus.cc:729] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3ff928ede147e68ca0f173e0bd323f, State: Initialized, Role: FOLLOWER
I20260620 14:35:11.998008 18929 consensus_queue.cc:260] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.998054 18931 raft_consensus.cc:348] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.998222 18931 raft_consensus.cc:374] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:11.998274 18929 ts_tablet_manager.cc:1428] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:11.998270 18931 raft_consensus.cc:729] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54371f65c2a24222bc577509c08b3a1c, State: Initialized, Role: FOLLOWER
I20260620 14:35:11.998318 18914 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 14:35:11.998401 18931 consensus_queue.cc:260] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:11.998664 18931 ts_tablet_manager.cc:1428] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:11.998669 18646 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
W20260620 14:35:12.049476 18781 tablet.cc:2369] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:12.141176 18936 raft_consensus.cc:482] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:12.141299 18936 raft_consensus.cc:504] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:12.141673 18936 leader_election.cc:288] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b5f83e50e4e4049a99c1ded451e3947 (127.0.98.2:42579), 2e3ff928ede147e68ca0f173e0bd323f (127.0.98.3:38623)
I20260620 14:35:12.144593 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.141885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42579 (local address 127.0.98.1:50557)
0620 14:35:12.141958 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.141972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.142035 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.142185 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.142186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.142189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.142270 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.142273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.142958 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.142959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.143763 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.143765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.144317 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.144318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.144413 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.144416 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.144458 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.144508 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":14}
I20260620 14:35:12.144603 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.141983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50557 (local address 127.0.98.2:42579)
0620 14:35:12.142028 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.142030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.142041 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:12.142102 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.142104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.142116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.142176 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.142347 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.142882 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.143868 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.144238 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.144291 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.144299 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.144486 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.144532 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:35:12.144640 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.142103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37245 (local address 127.0.98.3:38623)
0620 14:35:12.142152 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.142154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.142183 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:35:12.142241 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.142241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.142251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.142317 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.142503 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.142968 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.143908 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.144257 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.144344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.144353 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.144530 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.144570 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:35:12.144883 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.142017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38623 (local address 127.0.98.1:37245)
0620 14:35:12.142134 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.142143 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.142186 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.142358 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.142359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.142361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.142428 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.142430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.143046 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.143046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.143783 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.143784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.144371 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.144372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.144431 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.144435 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.144509 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.144544 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260620 14:35:12.144950 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b9dadf42a9470faa22a0d4f1b674cd" candidate_uuid: "54371f65c2a24222bc577509c08b3a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5f83e50e4e4049a99c1ded451e3947" is_pre_election: true
I20260620 14:35:12.145112 18734 raft_consensus.cc:2454] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54371f65c2a24222bc577509c08b3a1c in term 0.
I20260620 14:35:12.145377 18535 leader_election.cc:302] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5f83e50e4e4049a99c1ded451e3947, 54371f65c2a24222bc577509c08b3a1c; no voters:
I20260620 14:35:12.145516 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b9dadf42a9470faa22a0d4f1b674cd" candidate_uuid: "54371f65c2a24222bc577509c08b3a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3ff928ede147e68ca0f173e0bd323f" is_pre_election: true
I20260620 14:35:12.145539 18936 raft_consensus.cc:2781] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:12.145603 18936 raft_consensus.cc:482] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:12.145658 18936 raft_consensus.cc:3037] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.145680 18868 raft_consensus.cc:2454] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54371f65c2a24222bc577509c08b3a1c in term 0.
I20260620 14:35:12.146380 18936 raft_consensus.cc:504] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
I20260620 14:35:12.146507 18936 leader_election.cc:288] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [CANDIDATE]: Term 1 election: Requested vote from peers 1b5f83e50e4e4049a99c1ded451e3947 (127.0.98.2:42579), 2e3ff928ede147e68ca0f173e0bd323f (127.0.98.3:38623)
I20260620 14:35:12.146673 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b9dadf42a9470faa22a0d4f1b674cd" candidate_uuid: "54371f65c2a24222bc577509c08b3a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b5f83e50e4e4049a99c1ded451e3947"
I20260620 14:35:12.146675 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b9dadf42a9470faa22a0d4f1b674cd" candidate_uuid: "54371f65c2a24222bc577509c08b3a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3ff928ede147e68ca0f173e0bd323f"
I20260620 14:35:12.146754 18868 raft_consensus.cc:3037] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.146754 18734 raft_consensus.cc:3037] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.147632 18734 raft_consensus.cc:2454] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54371f65c2a24222bc577509c08b3a1c in term 1.
I20260620 14:35:12.147632 18868 raft_consensus.cc:2454] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54371f65c2a24222bc577509c08b3a1c in term 1.
I20260620 14:35:12.147791 18535 leader_election.cc:302] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3ff928ede147e68ca0f173e0bd323f, 54371f65c2a24222bc577509c08b3a1c; no voters:
I20260620 14:35:12.147879 18936 raft_consensus.cc:2781] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:12.148036 18936 raft_consensus.cc:686] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 1 LEADER]: Becoming Leader. State: Replica: 54371f65c2a24222bc577509c08b3a1c, State: Running, Role: LEADER
I20260620 14:35:12.148137 18936 consensus_queue.cc:237] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } }
W20260620 14:35:12.148449 18647 tablet.cc:2369] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:12.149004 18454 catalog_manager.cc:5495] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 54371f65c2a24222bc577509c08b3a1c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "54371f65c2a24222bc577509c08b3a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54371f65c2a24222bc577509c08b3a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20260620 14:35:12.169557 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.167048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.0.98.1:41451)
0620 14:35:12.167132 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.167135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.167144 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:12.167180 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.167181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.167202 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.167283 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:12.167425 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.167912 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.168729 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.168944 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.169026 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.169160 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:12.169259 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:12.169368 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.169441 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.169454 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:12.171346 18868 raft_consensus.cc:1261] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 1 FOLLOWER]: Refusing update from remote peer 54371f65c2a24222bc577509c08b3a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:12.171358 18734 raft_consensus.cc:1261] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 1 FOLLOWER]: Refusing update from remote peer 54371f65c2a24222bc577509c08b3a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:12.171720 18938 consensus_queue.cc:1035] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5f83e50e4e4049a99c1ded451e3947" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:12.171861 18936 consensus_queue.cc:1035] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3ff928ede147e68ca0f173e0bd323f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:12.174414 18942 mvcc.cc:205] Tried to move back new op lower bound from 7298933195450912768 to 7298933195359477760. Current Snapshot: MvccSnapshot[applied={T|T < 7298933195450912768}]
I20260620 14:35:12.174667 18943 mvcc.cc:205] Tried to move back new op lower bound from 7298933195450912768 to 7298933195359477760. Current Snapshot: MvccSnapshot[applied={T|T < 7298933195450912768 or (T in {7298933195450912768})}]
I20260620 14:35:12.176334 18453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:12.176433 18453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:12.177160 18453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 626095b0c34d486ca84d45634ab01007: Sending DeleteTablet for 3 replicas of tablet 96b9dadf42a9470faa22a0d4f1b674cd
I20260620 14:35:12.177485 18580 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38294
I20260620 14:35:12.177628 18848 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53936
I20260620 14:35:12.177609 18714 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b9dadf42a9470faa22a0d4f1b674cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49164
I20260620 14:35:12.177723 18950 tablet_replica.cc:330] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c: stopping tablet replica
I20260620 14:35:12.177839 18950 raft_consensus.cc:2229] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:35:12.177945 18950 raft_consensus.cc:2258] T 96b9dadf42a9470faa22a0d4f1b674cd P 54371f65c2a24222bc577509c08b3a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.178057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18517
I20260620 14:35:12.181385 18952 tablet_replica.cc:330] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: stopping tablet replica
I20260620 14:35:12.181526 18952 raft_consensus.cc:2229] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:12.181643 18952 raft_consensus.cc:2258] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.181856 18951 tablet_replica.cc:330] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: stopping tablet replica
I20260620 14:35:12.181954 18951 raft_consensus.cc:2229] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:12.182065 18951 raft_consensus.cc:2258] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.182214 18952 ts_tablet_manager.cc:1905] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:12.182602 18951 ts_tablet_manager.cc:1905] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:12.183774 18952 ts_tablet_manager.cc:1918] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:12.183862 18952 log.cc:1196] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/96b9dadf42a9470faa22a0d4f1b674cd
I20260620 14:35:12.184135 18952 ts_tablet_manager.cc:1939] T 96b9dadf42a9470faa22a0d4f1b674cd P 1b5f83e50e4e4049a99c1ded451e3947: Deleting consensus metadata
I20260620 14:35:12.184521 18951 ts_tablet_manager.cc:1918] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:12.184566 18441 catalog_manager.cc:4841] TS 1b5f83e50e4e4049a99c1ded451e3947 (127.0.98.2:42579): tablet 96b9dadf42a9470faa22a0d4f1b674cd (table test-table [id=ec3bd36054924830b13744173d478ae2]) successfully deleted
I20260620 14:35:12.184595 18951 log.cc:1196] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/96b9dadf42a9470faa22a0d4f1b674cd
I20260620 14:35:12.184862 18951 ts_tablet_manager.cc:1939] T 96b9dadf42a9470faa22a0d4f1b674cd P 2e3ff928ede147e68ca0f173e0bd323f: Deleting consensus metadata
I20260620 14:35:12.185225 18441 catalog_manager.cc:4841] TS 2e3ff928ede147e68ca0f173e0bd323f (127.0.98.3:38623): tablet 96b9dadf42a9470faa22a0d4f1b674cd (table test-table [id=ec3bd36054924830b13744173d478ae2]) successfully deleted
W20260620 14:35:12.185230 18439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41451 (error 108)
I20260620 14:35:12.185484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18651
I20260620 14:35:12.186074 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.185643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41451 (local address 127.0.0.1:38312)
0620 14:35:12.186002 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.186035 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
W20260620 14:35:12.186190 18439 catalog_manager.cc:4584] TS 54371f65c2a24222bc577509c08b3a1c (127.0.98.1:41451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96b9dadf42a9470faa22a0d4f1b674cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41451: connect: Connection refused (error 111)
I20260620 14:35:12.190387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18785
I20260620 14:35:12.195492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18423
2026-06-20T14:35:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2001 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:12Z Disabled control of system clock
I20260620 14:35:12.222519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42955
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:12.302630 18957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:12.302845 18957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:12.302865 18957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:12.304011 18957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:12.304051 18957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:12.304064 18957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:12.304076 18957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:12.304126 18957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:12.305456 18957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:12.305704 18957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:12.305981 18957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:12.310854 18957 server_base.cc:1034] running on GCE node
W20260620 14:35:12.310812 18965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.310853 18963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.310825 18962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:12.311275 18957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:12.311556 18957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:12.312695 18957 hybrid_clock.cc:648] HybridClock initialized: now 1781966112312684 us; error 30 us; skew 500 ppm
I20260620 14:35:12.313980 18957 webserver.cc:460] Webserver started at http://127.0.98.62:38653/ using document root <none> and password file <none>
I20260620 14:35:12.314215 18957 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:12.314261 18957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:12.314407 18957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:12.315440 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "f03cb0888b044e6d81fda8428916c2e2"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "6244903290fa1c796fdc8a0306210b99"
server_key_iv: "0716081f28124de5c5940495148257a3"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.315795 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "f03cb0888b044e6d81fda8428916c2e2"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "6244903290fa1c796fdc8a0306210b99"
server_key_iv: "0716081f28124de5c5940495148257a3"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.317078 18957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:12.317792 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.317951 18957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.318037 18957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "f03cb0888b044e6d81fda8428916c2e2"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "6244903290fa1c796fdc8a0306210b99"
server_key_iv: "0716081f28124de5c5940495148257a3"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.318112 18957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:12.337163 18957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:12.337517 18957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:12.337672 18957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:12.341941 18957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42955
I20260620 14:35:12.341996 19023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42955 every 8 connection(s)
I20260620 14:35:12.342322 18957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:12.343261 19024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:12.345471 19024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: Bootstrap starting.
I20260620 14:35:12.346022 19024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:12.346257 19024 log.cc:824] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:12.346822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 18957
I20260620 14:35:12.346931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:12.346964 19024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: No bootstrap required, opened a new log
I20260620 14:35:12.347155 392 external_mini_cluster.cc:1428] Setting key 486eba18bad0365345f6a0292c0b21b3
I20260620 14:35:12.349671 19024 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } }
I20260620 14:35:12.349841 19024 raft_consensus.cc:374] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:12.349885 19024 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03cb0888b044e6d81fda8428916c2e2, State: Initialized, Role: FOLLOWER
I20260620 14:35:12.350019 19024 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } }
I20260620 14:35:12.350097 19024 raft_consensus.cc:388] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:12.350124 19024 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:12.350171 19024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.350837 19024 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } }
I20260620 14:35:12.350961 19024 leader_election.cc:302] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f03cb0888b044e6d81fda8428916c2e2; no voters:
I20260620 14:35:12.351122 19024 leader_election.cc:288] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:12.351151 19029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:12.351251 19029 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [term 1 LEADER]: Becoming Leader. State: Replica: f03cb0888b044e6d81fda8428916c2e2, State: Running, Role: LEADER
I20260620 14:35:12.351338 19029 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } }
I20260620 14:35:12.351377 19024 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:12.351639 19029 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03cb0888b044e6d81fda8428916c2e2. Latest consensus state: current_term: 1 leader_uuid: "f03cb0888b044e6d81fda8428916c2e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } } }
I20260620 14:35:12.351652 19030 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f03cb0888b044e6d81fda8428916c2e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03cb0888b044e6d81fda8428916c2e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42955 } } }
I20260620 14:35:12.352092 19029 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:12.352113 19030 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:12.352351 19038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:12.352882 19038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:12.352962 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.347654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.0.98.62:42955)
0620 14:35:12.347848 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.347855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.347883 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:35:12.348343 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.348349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.348404 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.348512 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:12.348748 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.349611 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.350726 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.351287 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.351731 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.352145 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:12.352149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:12.352149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:12.352360 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:12.352402 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:12.352414 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:12.352498 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.352703 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.352782 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260620 14:35:12.354516 19038 catalog_manager.cc:1347] Generated new cluster ID: 4fb5c12b80654aaaa7f5eb1a3097d05e
I20260620 14:35:12.354569 19038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:12.382190 19038 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:12.382722 19038 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:12.390381 19038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: Generated new TSK 0
I20260620 14:35:12.390592 19038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:12.392910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--builtin_ntp_servers=127.0.98.20:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:12.473205 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:12.473400 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:12.473466 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:12.474594 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:12.474633 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:12.474658 19048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:12.476006 19048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:12.476235 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:12.476543 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:12.481361 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.481361 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.481513 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:12.481616 19048 server_base.cc:1034] running on GCE node
I20260620 14:35:12.481828 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:12.482080 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:12.483245 19048 hybrid_clock.cc:648] HybridClock initialized: now 1781966112483239 us; error 38 us; skew 500 ppm
I20260620 14:35:12.484467 19048 webserver.cc:460] Webserver started at http://127.0.98.1:39411/ using document root <none> and password file <none>
I20260620 14:35:12.484716 19048 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:12.484784 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:12.484941 19048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:12.486055 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "7ae7f00eee12be57de17a2992bc1da77"
server_key_iv: "a9a50c993d5d78724015db0ad8df33d5"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.486418 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "7ae7f00eee12be57de17a2992bc1da77"
server_key_iv: "a9a50c993d5d78724015db0ad8df33d5"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.487751 19048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:12.488445 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.488611 19048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:12.488713 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "7ae7f00eee12be57de17a2992bc1da77"
server_key_iv: "a9a50c993d5d78724015db0ad8df33d5"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.488790 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:12.497851 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:12.498116 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:12.498262 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:12.498483 19048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:12.498795 19048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:12.498854 19048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.498898 19048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:12.498924 19048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.504572 19048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43461
I20260620 14:35:12.504616 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43461 every 8 connection(s)
I20260620 14:35:12.504935 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:35:12.507557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19048
I20260620 14:35:12.507663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:12.507876 392 external_mini_cluster.cc:1428] Setting key 50cdda24c438947df43d88b301ebf05d
I20260620 14:35:12.508827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--builtin_ntp_servers=127.0.98.20:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:35:12.510565 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.505681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37493 (local address 127.0.98.62:42955)
0620 14:35:12.505724 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.505726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.505938 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:35:12.506468 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.506469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.506488 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.506540 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:12.507209 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.507825 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.509071 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.509291 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.509382 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.510320 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:12.510321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:12.510322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:12.510325 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:12.510339 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:12.510343 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:12.510386 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.510477 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.510508 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:12.510890 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.505641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42955 (local address 127.0.98.1:37493)
0620 14:35:12.505822 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.505854 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.506358 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.506610 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.506614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.506629 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:12.507114 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.507118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.507917 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.507918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.508981 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.508984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.509393 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.509395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.509580 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.510139 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:12.510140 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:12.510183 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:12.510216 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:12.510397 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:12.510401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:12.510404 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.510469 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.510576 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 14:35:12.511404 19176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42955
I20260620 14:35:12.511500 19176 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:12.511687 19176 heartbeater.cc:511] Master 127.0.98.62:42955 requested a full tablet report, sending...
I20260620 14:35:12.512166 18988 ts_manager.cc:194] Registered new tserver with Master: 16a622f6f1ee445ba3f4004cfe1a6fbf (127.0.98.1:43461)
I20260620 14:35:12.512849 18988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37493
W20260620 14:35:12.588816 19179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:12.589008 19179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:12.589030 19179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:12.590196 19179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:12.590236 19179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:12.590298 19179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:12.591940 19179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:12.592186 19179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:12.592491 19179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:12.597607 19184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:12.597627 19179 server_base.cc:1034] running on GCE node
W20260620 14:35:12.597596 19185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.597754 19187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:12.597970 19179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:12.598222 19179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:12.599386 19179 hybrid_clock.cc:648] HybridClock initialized: now 1781966112599373 us; error 27 us; skew 500 ppm
I20260620 14:35:12.603801 19179 webserver.cc:460] Webserver started at http://127.0.98.2:35175/ using document root <none> and password file <none>
I20260620 14:35:12.604054 19179 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:12.604127 19179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:12.604265 19179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:12.605301 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "381c78d59aa643249a363d4e154c5254"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "989a1c027ab989ca7001ff895760ae7b"
server_key_iv: "88dc9802b958e0bc90f74d1a2c608f7c"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.605702 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "381c78d59aa643249a363d4e154c5254"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "989a1c027ab989ca7001ff895760ae7b"
server_key_iv: "88dc9802b958e0bc90f74d1a2c608f7c"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.606998 19179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:12.607756 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.607939 19179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:12.608036 19179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "381c78d59aa643249a363d4e154c5254"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "989a1c027ab989ca7001ff895760ae7b"
server_key_iv: "88dc9802b958e0bc90f74d1a2c608f7c"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.608120 19179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:12.622102 19179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:12.622416 19179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:12.622584 19179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:12.622802 19179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:12.623122 19179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:12.623155 19179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.623198 19179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:12.623225 19179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.628909 19179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37295
I20260620 14:35:12.628968 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37295 every 8 connection(s)
I20260620 14:35:12.629304 19179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:35:12.634145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19179
I20260620 14:35:12.634255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:12.634518 392 external_mini_cluster.cc:1428] Setting key b2b036285093a3e05a2bd5a37d4a8451
I20260620 14:35:12.634611 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.630052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34407 (local address 127.0.98.62:42955)
0620 14:35:12.630118 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.630120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.630379 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:35:12.630847 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.630848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.630858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.630911 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:12.631573 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.632053 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.633034 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.633254 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.633348 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.634275 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:12.634277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:12.634278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:12.634283 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:12.634312 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:12.634317 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:12.634396 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.634493 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.634520 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:35:12.634794 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.630057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42955 (local address 127.0.98.2:34407)
0620 14:35:12.630259 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.630292 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.630767 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.630985 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.630988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.630996 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:12.631503 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.631508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.632150 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.632151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.632937 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.632939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.633360 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.633362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.633549 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.634108 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:12.634109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:12.634138 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:12.634160 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:12.634396 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:12.634401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:12.634404 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.634458 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.634540 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260620 14:35:12.635268 19307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42955
I20260620 14:35:12.635397 19307 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:12.635545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--builtin_ntp_servers=127.0.98.20:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:35:12.635568 19307 heartbeater.cc:511] Master 127.0.98.62:42955 requested a full tablet report, sending...
I20260620 14:35:12.635972 18987 ts_manager.cc:194] Registered new tserver with Master: 381c78d59aa643249a363d4e154c5254 (127.0.98.2:37295)
I20260620 14:35:12.636356 18987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34407
W20260620 14:35:12.716007 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:12.716205 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:12.716226 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:12.717360 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:12.717401 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:12.717480 19310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:12.718767 19310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:12.718947 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:12.719194 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:12.724273 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.724280 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:12.724303 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:12.724462 19310 server_base.cc:1034] running on GCE node
I20260620 14:35:12.724684 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:12.724920 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:12.726076 19310 hybrid_clock.cc:648] HybridClock initialized: now 1781966112726056 us; error 52 us; skew 500 ppm
I20260620 14:35:12.727305 19310 webserver.cc:460] Webserver started at http://127.0.98.3:40113/ using document root <none> and password file <none>
I20260620 14:35:12.727559 19310 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:12.727626 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:12.727761 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:12.728907 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d2b013455194f12940592a2dde8ab84"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "2876b115cee72396fb7d84d7e7933dad"
server_key_iv: "324204eb9391d64980a8d3efe4281959"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.729266 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d2b013455194f12940592a2dde8ab84"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "2876b115cee72396fb7d84d7e7933dad"
server_key_iv: "324204eb9391d64980a8d3efe4281959"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.730597 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:12.731272 19324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.731441 19310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:12.731536 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "0d2b013455194f12940592a2dde8ab84"
format_stamp: "Formatted at 2026-06-20 14:35:12 on dist-test-slave-tc2g"
server_key: "2876b115cee72396fb7d84d7e7933dad"
server_key_iv: "324204eb9391d64980a8d3efe4281959"
server_key_version: "encryptionkey@0"
I20260620 14:35:12.731617 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:12.741541 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:12.741824 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:12.741971 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:12.742194 19310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:12.742519 19310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:12.742580 19310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.742615 19310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:12.742643 19310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:12.748270 19310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34395
I20260620 14:35:12.748310 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34395 every 8 connection(s)
I20260620 14:35:12.748656 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:12.750654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19310
I20260620 14:35:12.750777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:35:12.751039 392 external_mini_cluster.cc:1428] Setting key 025c9b3fe4cd09bcd157aefdcdb91787
I20260620 14:35:12.754056 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.749369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45975 (local address 127.0.98.62:42955)
0620 14:35:12.749434 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.749436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.749662 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:35:12.750141 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.750142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.750152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.750208 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:12.750946 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.751681 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.752828 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.753018 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.753074 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.753837 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:12.753838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:12.753838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:12.753843 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:12.753856 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:12.753860 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:12.753915 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.753978 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.753998 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:35:12.754247 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.749334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42955 (local address 127.0.98.3:45975)
0620 14:35:12.749549 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.749589 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.750062 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.750262 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.750265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.750272 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:12.750825 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.750832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.751752 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.751753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.752755 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.752758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.753092 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.753093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.753205 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.753711 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:12.753711 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:12.753738 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:12.753760 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:12.753917 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:12.753919 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:12.753921 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.753969 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.754041 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260620 14:35:12.754678 19438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42955
I20260620 14:35:12.754760 19438 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:12.754967 19438 heartbeater.cc:511] Master 127.0.98.62:42955 requested a full tablet report, sending...
I20260620 14:35:12.755308 18987 ts_manager.cc:194] Registered new tserver with Master: 0d2b013455194f12940592a2dde8ab84 (127.0.98.3:34395)
I20260620 14:35:12.755669 18987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45975
I20260620 14:35:12.762851 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:12.767347 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.764208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.0.98.62:42955)
0620 14:35:12.764295 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.764297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.764406 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:35:12.764504 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.764505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.764514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.764581 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:12.764758 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.765234 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.766010 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.766204 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.766288 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.767045 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:12.767046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:12.767047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:12.767051 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:12.767066 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:12.767070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:12.767181 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.767254 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.767266 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:35:12.769160 18987 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:12.770792 18987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:12.776999 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.773194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.98.1:43461)
0620 14:35:12.773381 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.773386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.773436 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:35:12.773522 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.773524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.773664 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.773935 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.773985 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.774654 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.776021 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.776615 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.776703 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.776795 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.776876 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.776915 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 14:35:12.776986 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.773066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43461 (local address 127.0.0.1:41372)
0620 14:35:12.773359 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.773370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.773446 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.773775 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.773776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.773779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.773851 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.773853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.774747 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.774747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.775925 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.775927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.776705 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.776705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.776774 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.776786 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.776861 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.776915 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":92,"threads_started":1}
I20260620 14:35:12.777289 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.772844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37295 (local address 127.0.0.1:38360)
0620 14:35:12.773074 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.773117 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.773197 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.773766 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.773769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.773771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.773838 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.773841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.775205 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.775206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.776179 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.776181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.777080 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.777081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.777156 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.777162 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.777191 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.777225 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":23,"thread_start_us":60,"threads_started":1}
I20260620 14:35:12.777624 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.773251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34395 (local address 127.0.0.1:43350)
0620 14:35:12.773378 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.773389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.773445 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.773778 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.773778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.773780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.773838 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.773839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.775187 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.775188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.776212 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.776214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.777437 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.777438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.777491 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.777494 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.777524 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.777560 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":36,"thread_start_us":65,"threads_started":1}
I20260620 14:35:12.777745 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.773100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.98.2:37295)
0620 14:35:12.773575 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.773580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.773590 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:12.773634 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.773637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.773653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.773916 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.773977 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.775087 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.776266 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.776964 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.777478 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.777580 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.777604 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.777647 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":108,"threads_started":1}
I20260620 14:35:12.777735 19110 tablet_service.cc:1467] Processing CreateTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf (DEFAULT_TABLE table=test-table [id=7e4f447a56154c27a380c0c689962624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:12.778033 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.773459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.98.3:34395)
0620 14:35:12.773620 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.773624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.773633 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:12.773666 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.773668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.773684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.773916 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.773977 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.775087 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.776314 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.777017 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.777784 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.777889 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.777914 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.777953 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:35:12.778106 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a356d10f1db4f09bb45ee4854bf3bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:12.778383 19241 tablet_service.cc:1467] Processing CreateTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf (DEFAULT_TABLE table=test-table [id=7e4f447a56154c27a380c0c689962624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:12.778584 19372 tablet_service.cc:1467] Processing CreateTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf (DEFAULT_TABLE table=test-table [id=7e4f447a56154c27a380c0c689962624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:12.778723 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a356d10f1db4f09bb45ee4854bf3bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:12.778880 19372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a356d10f1db4f09bb45ee4854bf3bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:12.781148 19452 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: Bootstrap starting.
I20260620 14:35:12.781773 19452 tablet_bootstrap.cc:654] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:12.782071 19453 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Bootstrap starting.
I20260620 14:35:12.782068 19454 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Bootstrap starting.
I20260620 14:35:12.782130 19452 log.cc:824] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:12.782722 19453 tablet_bootstrap.cc:654] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:12.782722 19454 tablet_bootstrap.cc:654] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:12.782864 19452 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: No bootstrap required, opened a new log
I20260620 14:35:12.782930 19452 ts_tablet_manager.cc:1397] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:12.783061 19454 log.cc:824] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:12.783061 19453 log.cc:824] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:12.783771 19454 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: No bootstrap required, opened a new log
I20260620 14:35:12.783844 19454 ts_tablet_manager.cc:1397] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:12.784066 19453 tablet_bootstrap.cc:492] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: No bootstrap required, opened a new log
I20260620 14:35:12.784128 19453 ts_tablet_manager.cc:1397] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:12.785182 19452 raft_consensus.cc:348] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.785339 19452 raft_consensus.cc:374] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:12.785382 19452 raft_consensus.cc:729] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a622f6f1ee445ba3f4004cfe1a6fbf, State: Initialized, Role: FOLLOWER
I20260620 14:35:12.785532 19452 consensus_queue.cc:260] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.785789 19452 ts_tablet_manager.cc:1428] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:12.785929 19176 heartbeater.cc:503] Master 127.0.98.62:42955 was elected leader, sending a full tablet report...
I20260620 14:35:12.786513 19454 raft_consensus.cc:348] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.786677 19454 raft_consensus.cc:374] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:12.786720 19454 raft_consensus.cc:729] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381c78d59aa643249a363d4e154c5254, State: Initialized, Role: FOLLOWER
I20260620 14:35:12.786659 19453 raft_consensus.cc:348] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.786805 19453 raft_consensus.cc:374] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:12.786839 19453 raft_consensus.cc:729] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2b013455194f12940592a2dde8ab84, State: Initialized, Role: FOLLOWER
I20260620 14:35:12.786832 19454 consensus_queue.cc:260] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.786937 19453 consensus_queue.cc:260] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.787099 19454 ts_tablet_manager.cc:1428] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:12.787173 19453 ts_tablet_manager.cc:1428] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:12.787281 19307 heartbeater.cc:503] Master 127.0.98.62:42955 was elected leader, sending a full tablet report...
I20260620 14:35:12.787300 19438 heartbeater.cc:503] Master 127.0.98.62:42955 was elected leader, sending a full tablet report...
I20260620 14:35:12.864005 19458 raft_consensus.cc:482] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:12.864174 19458 raft_consensus.cc:504] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.864591 19458 leader_election.cc:288] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 381c78d59aa643249a363d4e154c5254 (127.0.98.2:37295), 0d2b013455194f12940592a2dde8ab84 (127.0.98.3:34395)
I20260620 14:35:12.867487 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.864845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40053 (local address 127.0.98.2:37295)
0620 14:35:12.864909 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.864911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.864922 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:12.864960 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.864961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.864970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.865038 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.865210 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.865764 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.866786 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.867143 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.867233 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.867243 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.867382 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.867421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:35:12.867491 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.864758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37295 (local address 127.0.98.1:40053)
0620 14:35:12.864816 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.864828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.864908 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.865067 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.865068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.865070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.865140 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.865142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.865868 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.865869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.866691 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.866694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.867235 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.867236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.867308 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.867311 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.867375 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.867418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":43}
I20260620 14:35:12.867565 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.864955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44787 (local address 127.0.98.3:34395)
0620 14:35:12.864996 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.864998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.865120 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:35:12.865188 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.865189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.865196 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.865251 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:12.865410 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.865894 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.866845 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.867202 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.867283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.867292 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.867456 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.867495 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:12.867619 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.864880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34395 (local address 127.0.98.1:44787)
0620 14:35:12.865052 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.865060 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:12.865132 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:12.865255 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:12.865256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:12.865257 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:12.865331 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.865333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.865988 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.865989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.866752 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:12.866753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.867285 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.867286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:12.867378 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.867381 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:12.867439 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:35:12.867477 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260620 14:35:12.867882 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a356d10f1db4f09bb45ee4854bf3bbf" candidate_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c78d59aa643249a363d4e154c5254" is_pre_election: true
I20260620 14:35:12.868005 19392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a356d10f1db4f09bb45ee4854bf3bbf" candidate_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2b013455194f12940592a2dde8ab84" is_pre_election: true
I20260620 14:35:12.868057 19261 raft_consensus.cc:2454] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a622f6f1ee445ba3f4004cfe1a6fbf in term 0.
I20260620 14:35:12.868155 19392 raft_consensus.cc:2454] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a622f6f1ee445ba3f4004cfe1a6fbf in term 0.
I20260620 14:35:12.868302 19063 leader_election.cc:302] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a622f6f1ee445ba3f4004cfe1a6fbf, 381c78d59aa643249a363d4e154c5254; no voters:
I20260620 14:35:12.868467 19458 raft_consensus.cc:2781] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:12.868546 19458 raft_consensus.cc:482] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:12.868562 19458 raft_consensus.cc:3037] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.869282 19458 raft_consensus.cc:504] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.869411 19458 leader_election.cc:288] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [CANDIDATE]: Term 1 election: Requested vote from peers 381c78d59aa643249a363d4e154c5254 (127.0.98.2:37295), 0d2b013455194f12940592a2dde8ab84 (127.0.98.3:34395)
I20260620 14:35:12.869570 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a356d10f1db4f09bb45ee4854bf3bbf" candidate_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c78d59aa643249a363d4e154c5254"
I20260620 14:35:12.869588 19392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a356d10f1db4f09bb45ee4854bf3bbf" candidate_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2b013455194f12940592a2dde8ab84"
I20260620 14:35:12.869663 19392 raft_consensus.cc:3037] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.869666 19261 raft_consensus.cc:3037] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:12.870371 19261 raft_consensus.cc:2454] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a622f6f1ee445ba3f4004cfe1a6fbf in term 1.
I20260620 14:35:12.870378 19392 raft_consensus.cc:2454] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a622f6f1ee445ba3f4004cfe1a6fbf in term 1.
I20260620 14:35:12.870535 19063 leader_election.cc:302] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a622f6f1ee445ba3f4004cfe1a6fbf, 381c78d59aa643249a363d4e154c5254; no voters:
I20260620 14:35:12.870625 19458 raft_consensus.cc:2781] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:12.870786 19458 raft_consensus.cc:686] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 1 LEADER]: Becoming Leader. State: Replica: 16a622f6f1ee445ba3f4004cfe1a6fbf, State: Running, Role: LEADER
I20260620 14:35:12.870900 19458 consensus_queue.cc:237] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } }
I20260620 14:35:12.871718 18987 catalog_manager.cc:5495] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a622f6f1ee445ba3f4004cfe1a6fbf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16a622f6f1ee445ba3f4004cfe1a6fbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 } health_report { overall_health: UNKNOWN } } }
W20260620 14:35:12.879841 19308 tablet.cc:2369] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:12.891649 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.889288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.98.1:43461)
0620 14:35:12.889373 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:12.889375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:12.889387 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:35:12.889466 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:12.889467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:12.889490 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:12.889560 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:12.889708 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.890166 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.890920 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:12.891111 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:12.891183 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:12.891324 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:12.891414 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:12.891516 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:12.891547 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 14:35:12.891557 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:35:12.893383 19261 raft_consensus.cc:1261] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 1 FOLLOWER]: Refusing update from remote peer 16a622f6f1ee445ba3f4004cfe1a6fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:12.893385 19392 raft_consensus.cc:1261] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 1 FOLLOWER]: Refusing update from remote peer 16a622f6f1ee445ba3f4004cfe1a6fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:12.893761 19458 consensus_queue.cc:1035] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c78d59aa643249a363d4e154c5254" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:12.893913 19462 consensus_queue.cc:1035] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2b013455194f12940592a2dde8ab84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:12.896528 19467 mvcc.cc:205] Tried to move back new op lower bound from 7298933198408708096 to 7298933198319931392. Current Snapshot: MvccSnapshot[applied={T|T < 7298933198408708096}]
I20260620 14:35:12.896763 19466 mvcc.cc:205] Tried to move back new op lower bound from 7298933198408708096 to 7298933198319931392. Current Snapshot: MvccSnapshot[applied={T|T < 7298933198408708096}]
I20260620 14:35:12.898404 18986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:12.898499 18986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:12.899264 18986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f03cb0888b044e6d81fda8428916c2e2: Sending DeleteTablet for 3 replicas of tablet 0a356d10f1db4f09bb45ee4854bf3bbf
I20260620 14:35:12.899626 19110 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='slave'} at 127.0.0.1:41372
I20260620 14:35:12.899726 19241 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='slave'} at 127.0.0.1:38360
I20260620 14:35:12.899856 19474 tablet_replica.cc:330] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf: stopping tablet replica
I20260620 14:35:12.899951 19474 raft_consensus.cc:2229] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:35:12.899950 19372 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a356d10f1db4f09bb45ee4854bf3bbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:12 UTC) from {username='slave'} at 127.0.0.1:43350
I20260620 14:35:12.900038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19048
I20260620 14:35:12.900041 19474 raft_consensus.cc:2258] T 0a356d10f1db4f09bb45ee4854bf3bbf P 16a622f6f1ee445ba3f4004cfe1a6fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.900620 19476 tablet_replica.cc:330] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: stopping tablet replica
I20260620 14:35:12.900709 19476 raft_consensus.cc:2229] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:12.900805 19476 raft_consensus.cc:2258] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.901175 19476 ts_tablet_manager.cc:1905] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:12.902679 19476 ts_tablet_manager.cc:1918] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:12.902741 19476 log.cc:1196] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/0a356d10f1db4f09bb45ee4854bf3bbf
I20260620 14:35:12.902982 19476 ts_tablet_manager.cc:1939] T 0a356d10f1db4f09bb45ee4854bf3bbf P 0d2b013455194f12940592a2dde8ab84: Deleting consensus metadata
I20260620 14:35:12.903345 18972 catalog_manager.cc:4841] TS 0d2b013455194f12940592a2dde8ab84 (127.0.98.3:34395): tablet 0a356d10f1db4f09bb45ee4854bf3bbf (table test-table [id=7e4f447a56154c27a380c0c689962624]) successfully deleted
I20260620 14:35:12.905532 19475 tablet_replica.cc:330] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254: stopping tablet replica
I20260620 14:35:12.905629 19475 raft_consensus.cc:2229] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:35:12.905840 18972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43461 (error 108)
I20260620 14:35:12.906025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19179
I20260620 14:35:12.906155 19475 raft_consensus.cc:2258] T 0a356d10f1db4f09bb45ee4854bf3bbf P 381c78d59aa643249a363d4e154c5254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:12.906286 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.906168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43461 (local address 127.0.0.1:41400)
0620 14:35:12.906214 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.906248 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:35:12.906397 18972 catalog_manager.cc:4584] TS 16a622f6f1ee445ba3f4004cfe1a6fbf (127.0.98.1:43461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a356d10f1db4f09bb45ee4854bf3bbf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43461: connect: Connection refused (error 111)
I20260620 14:35:12.910640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19310
I20260620 14:35:12.910671 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:12.910540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37295 (local address 127.0.0.1:38368)
0620 14:35:12.910619 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:12.910647 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:35:12.914925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 18957
2026-06-20T14:35:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (719 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:12 dist-test-slave-tc2g krb5kdc[19480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:12 dist-test-slave-tc2g krb5kdc[19480](info): set up 2 sockets
Jun 20 14:35:12 dist-test-slave-tc2g krb5kdc[19480](info): commencing operation
krb5kdc: starting...
W20260620 14:35:13.949361 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:14.066430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:14.145768 19493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:14.145989 19493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:14.146013 19493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:14.147222 19493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:14.147267 19493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:14.147282 19493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:14.147336 19493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:14.147362 19493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:14.148923 19493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:14.149196 19493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:14.149464 19493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:14.154352 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:14.154328 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.154513 19493 server_base.cc:1034] running on GCE node
W20260620 14:35:14.154331 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.154835 19493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:14.155126 19493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:14.156292 19493 hybrid_clock.cc:648] HybridClock initialized: now 1781966114156273 us; error 33 us; skew 500 ppm
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:14.158921 19493 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:14.159421 19493 webserver.cc:460] Webserver started at http://127.0.98.62:45567/ using document root <none> and password file <none>
I20260620 14:35:14.159605 19493 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:14.159655 19493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:14.159747 19493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:14.160758 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "58d5124f7e3a459788a820dce3691a8c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "4129a3c4396c1276d06b5ae11882d09b"
server_key_iv: "6493e105f05114750afcf0fb9b344c3a"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.161067 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "58d5124f7e3a459788a820dce3691a8c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "4129a3c4396c1276d06b5ae11882d09b"
server_key_iv: "6493e105f05114750afcf0fb9b344c3a"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.162443 19493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:14.163125 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.163314 19493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:14.163388 19493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "58d5124f7e3a459788a820dce3691a8c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "4129a3c4396c1276d06b5ae11882d09b"
server_key_iv: "6493e105f05114750afcf0fb9b344c3a"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.163440 19493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:14.174415 19493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:14.176088 19493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:14.176201 19493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:14.179953 19493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43433
I20260620 14:35:14.179980 19560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43433 every 8 connection(s)
I20260620 14:35:14.180343 19493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:14.180785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19493
I20260620 14:35:14.180877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:14.181088 392 external_mini_cluster.cc:1428] Setting key 6b0389ee1346385cfa4170cb32a8fab1
I20260620 14:35:14.181344 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:14.184082 19561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: Bootstrap starting.
I20260620 14:35:14.184688 19561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:14.185045 19561 log.cc:824] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:14.185786 19561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: No bootstrap required, opened a new log
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:14.188160 19561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } }
I20260620 14:35:14.188300 19561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:14.188324 19561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d5124f7e3a459788a820dce3691a8c, State: Initialized, Role: FOLLOWER
I20260620 14:35:14.188517 19561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } }
I20260620 14:35:14.188616 19561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:14.188652 19561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:14.188721 19561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:14.189476 19561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } }
I20260620 14:35:14.189606 19561 leader_election.cc:302] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58d5124f7e3a459788a820dce3691a8c; no voters:
I20260620 14:35:14.189785 19561 leader_election.cc:288] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:14.189849 19566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:14.189994 19566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [term 1 LEADER]: Becoming Leader. State: Replica: 58d5124f7e3a459788a820dce3691a8c, State: Running, Role: LEADER
I20260620 14:35:14.190056 19561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:14.190095 19566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } }
I20260620 14:35:14.190408 19566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58d5124f7e3a459788a820dce3691a8c. Latest consensus state: current_term: 1 leader_uuid: "58d5124f7e3a459788a820dce3691a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } } }
I20260620 14:35:14.190493 19566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:14.190414 19567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58d5124f7e3a459788a820dce3691a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5124f7e3a459788a820dce3691a8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43433 } } }
I20260620 14:35:14.190968 19567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:14.190697 19572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:14.191555 19572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:14.192673 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.181588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.98.62:43433)
0620 14:35:14.181758 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.181768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.181801 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:35:14.182402 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.182409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.182455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.182598 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:14.183228 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.184089 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.184892 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.185495 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.185611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.188433 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:14.188437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:14.188437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:14.188449 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:14.190501 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.190950 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.190953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.190954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.191065 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.191320 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.191321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.191322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.191655 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:14.192268 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.192338 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.192469 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260620 14:35:14.193085 19572 catalog_manager.cc:1347] Generated new cluster ID: a633db61e371442ca30fba42307672f5
I20260620 14:35:14.193154 19572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:14.209365 19572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:14.209913 19572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:14.213044 19572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: Generated new TSK 0
I20260620 14:35:14.213227 19572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:35:14.245790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:14.325598 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:14.325798 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:14.325820 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:14.326957 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:14.327020 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:14.327046 19587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:14.328531 19587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:14.328783 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:14.328997 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:14.333868 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:14.333905 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.334079 19587 server_base.cc:1034] running on GCE node
W20260620 14:35:14.333905 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.334328 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:14.334554 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:14.335726 19587 hybrid_clock.cc:648] HybridClock initialized: now 1781966114335711 us; error 28 us; skew 500 ppm
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:14.338196 19587 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:35:14.338624 19587 webserver.cc:460] Webserver started at http://127.0.98.1:45367/ using document root <none> and password file <none>
I20260620 14:35:14.338860 19587 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:14.338927 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:14.339071 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:14.340109 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "d941a270d2f246c4b5fcea689f2fef0c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "950416c48bf64486c5d38f921a95a003"
server_key_iv: "997041a01312c1dcd5a92bb67a5adb04"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.340449 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d941a270d2f246c4b5fcea689f2fef0c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "950416c48bf64486c5d38f921a95a003"
server_key_iv: "997041a01312c1dcd5a92bb67a5adb04"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.341742 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:35:14.342439 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.342607 19587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:14.342734 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "d941a270d2f246c4b5fcea689f2fef0c"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "950416c48bf64486c5d38f921a95a003"
server_key_iv: "997041a01312c1dcd5a92bb67a5adb04"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.342818 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:14.359625 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:14.361534 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:14.361709 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:14.361943 19587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:14.362346 19587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:14.362408 19587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.362449 19587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:14.362484 19587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.368413 19587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43747
I20260620 14:35:14.368467 19715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43747 every 8 connection(s)
I20260620 14:35:14.368821 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:35:14.370507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19587
I20260620 14:35:14.370671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:14.370901 392 external_mini_cluster.cc:1428] Setting key bf2e3ceea1dc6eaceff9a5b830bf8a29
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:35:14.379341 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.369598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38993 (local address 127.0.98.62:43433)
0620 14:35:14.369639 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.369641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.369848 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:35:14.371133 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.371135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.371148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.371201 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:14.372044 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.372531 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.373783 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.374119 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.374194 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.376337 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:14.376339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:14.376339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:14.376348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:14.377956 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.378338 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.378339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.378340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.378405 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.378649 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.378650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.378651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.378799 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:14.378878 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.379118 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.379206 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:14.379535 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.369586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43433 (local address 127.0.98.1:38993)
0620 14:35:14.369730 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.369766 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.370990 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.371275 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.371279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.371642 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:14.371953 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.371960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.372624 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.372625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.373702 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.373705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.374214 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.374214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.374330 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.374884 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:14.374892 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:14.376245 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:14.378035 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.378041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.378043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.378257 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.378477 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.378478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.378478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.378580 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.378877 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:14.378882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:14.378995 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.379075 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.379192 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260620 14:35:14.380138 19716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43433
I20260620 14:35:14.380247 19716 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:14.380427 19716 heartbeater.cc:511] Master 127.0.98.62:43433 requested a full tablet report, sending...
I20260620 14:35:14.380913 19525 ts_manager.cc:194] Registered new tserver with Master: d941a270d2f246c4b5fcea689f2fef0c (127.0.98.1:43747)
I20260620 14:35:14.381672 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:35:14.396644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:14.476639 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:14.476842 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:14.476866 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:14.478016 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:14.478050 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:14.478075 19721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:14.479465 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:14.479646 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:14.479812 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:14.484645 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.484673 19721 server_base.cc:1034] running on GCE node
W20260620 14:35:14.484683 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:14.484645 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.485097 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:14.485282 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:14.486455 19721 hybrid_clock.cc:648] HybridClock initialized: now 1781966114486433 us; error 36 us; skew 500 ppm
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:14.488890 19721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:35:14.489470 19721 webserver.cc:460] Webserver started at http://127.0.98.2:40653/ using document root <none> and password file <none>
I20260620 14:35:14.489706 19721 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:14.489770 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:14.489902 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:14.491087 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "945b11edffb1496b94319cc3f1b6f9b8"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "dcccc3d7422dd63ada2d49e4c804ebc6"
server_key_iv: "dad94078921a466c552eaf35ed2c02d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.491523 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "945b11edffb1496b94319cc3f1b6f9b8"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "dcccc3d7422dd63ada2d49e4c804ebc6"
server_key_iv: "dad94078921a466c552eaf35ed2c02d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.492857 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:14.493848 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.494019 19721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:14.494114 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "945b11edffb1496b94319cc3f1b6f9b8"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "dcccc3d7422dd63ada2d49e4c804ebc6"
server_key_iv: "dad94078921a466c552eaf35ed2c02d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.494199 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:14.515051 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:14.516832 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:14.517009 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:14.517256 19721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:14.517630 19721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:14.517685 19721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.517728 19721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:14.517757 19721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.523695 19721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34025
I20260620 14:35:14.523737 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34025 every 8 connection(s)
I20260620 14:35:14.524041 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:14.531315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19721
I20260620 14:35:14.531442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:14.531639 392 external_mini_cluster.cc:1428] Setting key f6e6e9fd6807fc10f00763cee22ec1ec
I20260620 14:35:14.533524 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.524675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55297 (local address 127.0.98.62:43433)
0620 14:35:14.524754 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.524757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.524940 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:35:14.525987 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.525988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.525998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.526046 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:14.526617 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.527105 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.528057 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.528231 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.528281 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.530290 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:14.530292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:14.530293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:14.530319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:14.531962 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.532430 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.532431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.532432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.532494 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.532728 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.532730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.532730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.532884 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:14.532950 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.533271 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.533356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:35:14.533810 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.524631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43433 (local address 127.0.98.2:55297)
0620 14:35:14.524795 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.524830 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.525891 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.526107 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.526110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.526364 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:14.526546 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.526550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.527180 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.527181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.527975 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.527977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.528301 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.528302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.528399 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.528832 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:14.528839 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:14.530195 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:14.532045 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.532053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.532055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.532341 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.532562 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.532563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.532563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.532663 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.532982 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:14.532988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:14.533142 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.533226 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.533345 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260620 14:35:14.534426 19850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43433
I20260620 14:35:14.534518 19850 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:14.534703 19850 heartbeater.cc:511] Master 127.0.98.62:43433 requested a full tablet report, sending...
I20260620 14:35:14.535081 19525 ts_manager.cc:194] Registered new tserver with Master: 945b11edffb1496b94319cc3f1b6f9b8 (127.0.98.2:34025)
I20260620 14:35:14.535614 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55297
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:35:14.555958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:14.637053 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:14.637247 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:14.637270 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:14.638393 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:14.638433 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:14.638458 19855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:14.639863 19855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:14.640039 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:14.640224 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:14.645087 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:14.645080 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.645328 19855 server_base.cc:1034] running on GCE node
W20260620 14:35:14.645143 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:14.645668 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:14.645926 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:14.647090 19855 hybrid_clock.cc:648] HybridClock initialized: now 1781966114647082 us; error 27 us; skew 500 ppm
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:14.649577 19855 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:35:14.650058 19855 webserver.cc:460] Webserver started at http://127.0.98.3:34873/ using document root <none> and password file <none>
I20260620 14:35:14.650326 19855 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:14.650399 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:14.650542 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:14.651561 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "e779e27ae1c14c109edbc2dbe6b59734"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "a6a92368e8f8859d8b562187f476bdc4"
server_key_iv: "7a564a0ac7f814075dd3e263dce76b4f"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.651919 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e779e27ae1c14c109edbc2dbe6b59734"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "a6a92368e8f8859d8b562187f476bdc4"
server_key_iv: "7a564a0ac7f814075dd3e263dce76b4f"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.653203 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:14.653980 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.654158 19855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.654255 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "e779e27ae1c14c109edbc2dbe6b59734"
format_stamp: "Formatted at 2026-06-20 14:35:14 on dist-test-slave-tc2g"
server_key: "a6a92368e8f8859d8b562187f476bdc4"
server_key_iv: "7a564a0ac7f814075dd3e263dce76b4f"
server_key_version: "encryptionkey@0"
I20260620 14:35:14.654343 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:14.667021 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:14.668797 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:14.668970 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:14.669191 19855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:14.669557 19855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:14.669610 19855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.669652 19855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:14.669677 19855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:14.675572 19855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40529
I20260620 14:35:14.675623 19983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40529 every 8 connection(s)
I20260620 14:35:14.675949 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:14.680546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 19855
I20260620 14:35:14.680651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:35:14.680876 392 external_mini_cluster.cc:1428] Setting key 8c830942c2d2afb7a17c0badde5c97ee
Jun 20 14:35:14 dist-test-slave-tc2g krb5kdc[19480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:14.685374 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.676642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49145 (local address 127.0.98.62:43433)
0620 14:35:14.676714 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.676716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.676908 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:35:14.677966 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.677967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.677977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.678040 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:14.678621 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.679115 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.680245 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.680585 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.680664 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.683093 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:14.683095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:14.683095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:14.683102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:14.684243 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.684591 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.684591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.684592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.684640 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.684862 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.684862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.684862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.684960 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:14.685011 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.685245 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.685308 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:35:14.685643 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.676611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43433 (local address 127.0.98.3:49145)
0620 14:35:14.676779 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.676821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.677864 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.678097 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.678100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.678361 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:14.678549 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.678553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.679192 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.679192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.680163 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.680166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.680739 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.680740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.680913 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.681478 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:14.681490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:14.683009 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:14.684313 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.684319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.684321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.684526 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.684696 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:14.684696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:14.684696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:14.684798 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:14.685013 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:14.685017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:14.685125 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.685221 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.685342 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260620 14:35:14.686121 19984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43433
I20260620 14:35:14.686199 19984 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:14.686339 19984 heartbeater.cc:511] Master 127.0.98.62:43433 requested a full tablet report, sending...
I20260620 14:35:14.686671 19525 ts_manager.cc:194] Registered new tserver with Master: e779e27ae1c14c109edbc2dbe6b59734 (127.0.98.3:40529)
I20260620 14:35:14.687016 19525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49145
I20260620 14:35:14.692993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:14.700052 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.694322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.0.98.62:43433)
0620 14:35:14.694395 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.694397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.694442 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:35:14.694500 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.694500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.694509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.694556 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:14.695075 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.695534 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.696253 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.696425 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.696473 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.697705 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:14.697706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:14.697707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:14.697714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:14.699122 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.699404 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.699405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.699405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.699451 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:14.699598 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:14.699598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:14.699598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:14.699715 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:14.699785 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.699892 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.699950 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:14.701474 19525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:14.703016 19525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:14.709399 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.0.98.2:34025)
0620 14:35:14.705665 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.705670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.705740 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:35:14.705830 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.705832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.705860 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.705935 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:14.706178 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.706882 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.708309 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.709005 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.709073 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709126 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.709276 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.709317 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260620 14:35:14.709414 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43747 (local address 127.0.0.1:45842)
0620 14:35:14.705237 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.705277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.705342 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.705675 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.705678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.705680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:14.705750 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.705752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.707038 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.707039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.708052 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.708054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.709057 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.709058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.709137 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709150 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.709260 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.709313 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260620 14:35:14.709657 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.0.98.3:40529)
0620 14:35:14.705525 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.705529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.705894 (+ 365us) server_negotiation.cc:407] Connection header received
0620 14:35:14.706104 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.706107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.706123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.706193 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:14.706379 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.707394 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.708588 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.709107 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.709174 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709234 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.709473 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.709583 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260620 14:35:14.709722 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.1:43747)
0620 14:35:14.705494 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.705499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.705510 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:14.705553 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.705555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.705572 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.705670 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:14.705827 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.706923 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.708141 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.708886 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.708957 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709019 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.709610 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.709646 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":115,"threads_started":1}
I20260620 14:35:14.709441 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34025 (local address 127.0.0.1:58876)
0620 14:35:14.705687 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.705696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.705754 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.706020 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.706020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.706022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:14.706092 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.706093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.706987 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.706988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.708190 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.708192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.709092 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.709094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.709165 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709169 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.709224 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.709288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":38,"thread_start_us":61,"threads_started":1}
I20260620 14:35:14.710086 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.705092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40529 (local address 127.0.0.1:59522)
0620 14:35:14.705834 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.705843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.705891 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.706272 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.706273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.706275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:14.706311 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.706313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.707500 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.707501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.708508 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.708509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.709265 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.709266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.709316 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.709320 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.709380 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.709414 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":49,"threads_started":1}
I20260620 14:35:14.710428 19650 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3a621e232421aa3a13213db373cb8 (DEFAULT_TABLE table=test-table [id=b94ed05234784dca8ee59c057baa7bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:14.710793 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3a621e232421aa3a13213db373cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:14.710736 19784 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3a621e232421aa3a13213db373cb8 (DEFAULT_TABLE table=test-table [id=b94ed05234784dca8ee59c057baa7bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:14.711050 19784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3a621e232421aa3a13213db373cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:14.714298 19999 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Bootstrap starting.
I20260620 14:35:14.714290 19998 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Bootstrap starting.
I20260620 14:35:14.714684 19918 tablet_service.cc:1467] Processing CreateTablet for tablet 75f3a621e232421aa3a13213db373cb8 (DEFAULT_TABLE table=test-table [id=b94ed05234784dca8ee59c057baa7bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:14.714990 19918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f3a621e232421aa3a13213db373cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:14.715047 19999 tablet_bootstrap.cc:654] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:14.715111 19998 tablet_bootstrap.cc:654] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:14.715688 19999 log.cc:824] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:14.715687 19998 log.cc:824] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:14.716487 19998 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: No bootstrap required, opened a new log
I20260620 14:35:14.716557 19998 ts_tablet_manager.cc:1397] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:14.716692 19999 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: No bootstrap required, opened a new log
I20260620 14:35:14.716739 19999 ts_tablet_manager.cc:1397] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:35:14.717964 20002 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Bootstrap starting.
I20260620 14:35:14.718556 20002 tablet_bootstrap.cc:654] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:14.718580 19998 raft_consensus.cc:348] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.718668 19999 raft_consensus.cc:348] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.718770 19998 raft_consensus.cc:374] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:14.718814 19999 raft_consensus.cc:374] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:14.718820 19998 raft_consensus.cc:729] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945b11edffb1496b94319cc3f1b6f9b8, State: Initialized, Role: FOLLOWER
I20260620 14:35:14.718850 19999 raft_consensus.cc:729] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d941a270d2f246c4b5fcea689f2fef0c, State: Initialized, Role: FOLLOWER
I20260620 14:35:14.718945 19999 consensus_queue.cc:260] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.718988 19998 consensus_queue.cc:260] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.719159 20002 log.cc:824] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:14.719236 19999 ts_tablet_manager.cc:1428] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:35:14.719236 19998 ts_tablet_manager.cc:1428] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:14.719348 19850 heartbeater.cc:503] Master 127.0.98.62:43433 was elected leader, sending a full tablet report...
I20260620 14:35:14.719368 19716 heartbeater.cc:503] Master 127.0.98.62:43433 was elected leader, sending a full tablet report...
I20260620 14:35:14.719969 20002 tablet_bootstrap.cc:492] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: No bootstrap required, opened a new log
I20260620 14:35:14.720055 20002 ts_tablet_manager.cc:1397] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:14.721860 20002 raft_consensus.cc:348] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.722013 20002 raft_consensus.cc:374] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:14.722059 20002 raft_consensus.cc:729] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e779e27ae1c14c109edbc2dbe6b59734, State: Initialized, Role: FOLLOWER
I20260620 14:35:14.722179 20002 consensus_queue.cc:260] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.722424 20002 ts_tablet_manager.cc:1428] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:35:14.722465 19984 heartbeater.cc:503] Master 127.0.98.62:43433 was elected leader, sending a full tablet report...
W20260620 14:35:14.774550 19851 tablet.cc:2369] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:14.841532 20003 raft_consensus.cc:482] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:14.841656 20003 raft_consensus.cc:504] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.842058 20003 leader_election.cc:288] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 945b11edffb1496b94319cc3f1b6f9b8 (127.0.98.2:34025), e779e27ae1c14c109edbc2dbe6b59734 (127.0.98.3:40529)
I20260620 14:35:14.846102 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.842378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34599 (local address 127.0.98.3:40529)
0620 14:35:14.842471 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.842473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.842483 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:14.842535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.842536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.842548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.842617 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:14.842834 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.843585 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845109 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.845725 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.845816 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.845988 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.846031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:35:14.846103 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.842344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43493 (local address 127.0.98.2:34025)
0620 14:35:14.842393 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.842395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.842534 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:35:14.842563 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.842564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.842576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.842632 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:14.842832 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.843579 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845160 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.845742 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845805 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.845817 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.845989 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.846032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:35:14.846129 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.842236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40529 (local address 127.0.98.1:34599)
0620 14:35:14.842300 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.842313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.842365 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.842650 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.842651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.842653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:14.842753 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.842755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.843669 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.843670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.845015 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.845019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.845819 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.845902 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.845908 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.845971 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.846027 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":61}
I20260620 14:35:14.846135 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.842250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34025 (local address 127.0.98.1:43493)
0620 14:35:14.842457 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.842473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:14.842516 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:14.842650 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:14.842651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:14.842653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:14.842759 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.842761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.843669 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.843670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.845040 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:14.845042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.845819 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.845820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:14.845902 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.845908 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:35:14.845971 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:35:14.846026 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":147,"thread_start_us":122,"threads_started":1}
I20260620 14:35:14.846489 19938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3a621e232421aa3a13213db373cb8" candidate_uuid: "d941a270d2f246c4b5fcea689f2fef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e779e27ae1c14c109edbc2dbe6b59734" is_pre_election: true
I20260620 14:35:14.846655 19938 raft_consensus.cc:2454] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d941a270d2f246c4b5fcea689f2fef0c in term 0.
I20260620 14:35:14.846755 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3a621e232421aa3a13213db373cb8" candidate_uuid: "d941a270d2f246c4b5fcea689f2fef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945b11edffb1496b94319cc3f1b6f9b8" is_pre_election: true
I20260620 14:35:14.846915 19804 raft_consensus.cc:2454] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d941a270d2f246c4b5fcea689f2fef0c in term 0.
I20260620 14:35:14.846925 19604 leader_election.cc:302] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d941a270d2f246c4b5fcea689f2fef0c, e779e27ae1c14c109edbc2dbe6b59734; no voters:
I20260620 14:35:14.847080 20003 raft_consensus.cc:2781] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:14.847128 20003 raft_consensus.cc:482] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:14.847144 20003 raft_consensus.cc:3037] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:14.847856 20003 raft_consensus.cc:504] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.847998 20003 leader_election.cc:288] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [CANDIDATE]: Term 1 election: Requested vote from peers 945b11edffb1496b94319cc3f1b6f9b8 (127.0.98.2:34025), e779e27ae1c14c109edbc2dbe6b59734 (127.0.98.3:40529)
I20260620 14:35:14.848163 19938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3a621e232421aa3a13213db373cb8" candidate_uuid: "d941a270d2f246c4b5fcea689f2fef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e779e27ae1c14c109edbc2dbe6b59734"
I20260620 14:35:14.848163 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f3a621e232421aa3a13213db373cb8" candidate_uuid: "d941a270d2f246c4b5fcea689f2fef0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945b11edffb1496b94319cc3f1b6f9b8"
I20260620 14:35:14.848245 19938 raft_consensus.cc:3037] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:14.848245 19804 raft_consensus.cc:3037] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:14.849121 19804 raft_consensus.cc:2454] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d941a270d2f246c4b5fcea689f2fef0c in term 1.
I20260620 14:35:14.849121 19938 raft_consensus.cc:2454] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d941a270d2f246c4b5fcea689f2fef0c in term 1.
I20260620 14:35:14.849275 19604 leader_election.cc:302] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d941a270d2f246c4b5fcea689f2fef0c, e779e27ae1c14c109edbc2dbe6b59734; no voters:
I20260620 14:35:14.849380 20003 raft_consensus.cc:2781] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:14.849550 20003 raft_consensus.cc:686] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [term 1 LEADER]: Becoming Leader. State: Replica: d941a270d2f246c4b5fcea689f2fef0c, State: Running, Role: LEADER
I20260620 14:35:14.849668 20003 consensus_queue.cc:237] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } }
I20260620 14:35:14.850483 19524 catalog_manager.cc:5495] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c reported cstate change: term changed from 0 to 1, leader changed from <none> to d941a270d2f246c4b5fcea689f2fef0c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d941a270d2f246c4b5fcea689f2fef0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d941a270d2f246c4b5fcea689f2fef0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 } health_report { overall_health: UNKNOWN } } }
W20260620 14:35:14.869602 19717 tablet.cc:2369] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:14.890406 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.887726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.0.98.1:43747)
0620 14:35:14.887786 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:14.887788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:14.887798 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:14.887882 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:14.887883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:14.887911 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:14.887996 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:14.888147 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.888621 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.889407 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:14.889641 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:14.889720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:14.889891 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:14.890058 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:14.890229 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:14.890250 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 14:35:14.890270 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:35:14.892326 19804 raft_consensus.cc:1261] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 1 FOLLOWER]: Refusing update from remote peer d941a270d2f246c4b5fcea689f2fef0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:14.892355 19938 raft_consensus.cc:1261] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 1 FOLLOWER]: Refusing update from remote peer d941a270d2f246c4b5fcea689f2fef0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:14.892735 20008 consensus_queue.cc:1035] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e779e27ae1c14c109edbc2dbe6b59734" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:14.892876 20003 consensus_queue.cc:1035] T 75f3a621e232421aa3a13213db373cb8 P d941a270d2f246c4b5fcea689f2fef0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "945b11edffb1496b94319cc3f1b6f9b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:14.896247 20012 mvcc.cc:205] Tried to move back new op lower bound from 7298933206596247552 to 7298933206424973312. Current Snapshot: MvccSnapshot[applied={T|T < 7298933206596247552 or (T in {7298933206596247552})}]
I20260620 14:35:14.896389 20013 mvcc.cc:205] Tried to move back new op lower bound from 7298933206596247552 to 7298933206424973312. Current Snapshot: MvccSnapshot[applied={T|T < 7298933206596247552 or (T in {7298933206596247552})}]
I20260620 14:35:14.897653 19524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:14.897768 19524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:14.898510 19524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58d5124f7e3a459788a820dce3691a8c: Sending DeleteTablet for 3 replicas of tablet 75f3a621e232421aa3a13213db373cb8
I20260620 14:35:14.898857 19784 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3a621e232421aa3a13213db373cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58876
I20260620 14:35:14.898881 19918 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3a621e232421aa3a13213db373cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59522
I20260620 14:35:14.898931 19650 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f3a621e232421aa3a13213db373cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45842
I20260620 14:35:14.899129 20020 tablet_replica.cc:330] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: stopping tablet replica
I20260620 14:35:14.899130 20021 tablet_replica.cc:330] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: stopping tablet replica
I20260620 14:35:14.899228 20020 raft_consensus.cc:2229] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:14.899291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19587
I20260620 14:35:14.899369 20020 raft_consensus.cc:2258] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:14.899274 20021 raft_consensus.cc:2229] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:14.899549 20021 raft_consensus.cc:2258] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:14.900247 20021 ts_tablet_manager.cc:1905] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:14.900789 20020 ts_tablet_manager.cc:1905] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:14.902022 20021 ts_tablet_manager.cc:1918] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:14.902101 20021 log.cc:1196] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/75f3a621e232421aa3a13213db373cb8
I20260620 14:35:14.902375 20021 ts_tablet_manager.cc:1939] T 75f3a621e232421aa3a13213db373cb8 P e779e27ae1c14c109edbc2dbe6b59734: Deleting consensus metadata
I20260620 14:35:14.902715 20020 ts_tablet_manager.cc:1918] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:14.902784 20020 log.cc:1196] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/75f3a621e232421aa3a13213db373cb8
I20260620 14:35:14.902851 19510 catalog_manager.cc:4841] TS e779e27ae1c14c109edbc2dbe6b59734 (127.0.98.3:40529): tablet 75f3a621e232421aa3a13213db373cb8 (table test-table [id=b94ed05234784dca8ee59c057baa7bdf]) successfully deleted
I20260620 14:35:14.903051 20020 ts_tablet_manager.cc:1939] T 75f3a621e232421aa3a13213db373cb8 P 945b11edffb1496b94319cc3f1b6f9b8: Deleting consensus metadata
I20260620 14:35:14.903440 19511 catalog_manager.cc:4841] TS 945b11edffb1496b94319cc3f1b6f9b8 (127.0.98.2:34025): tablet 75f3a621e232421aa3a13213db373cb8 (table test-table [id=b94ed05234784dca8ee59c057baa7bdf]) successfully deleted
W20260620 14:35:14.905138 19509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43747 (error 108)
I20260620 14:35:14.905398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19721
I20260620 14:35:14.905625 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:14.905491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43747 (local address 127.0.0.1:45858)
0620 14:35:14.905539 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:14.905586 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:35:14.906010 19509 catalog_manager.cc:4584] TS d941a270d2f246c4b5fcea689f2fef0c (127.0.98.1:43747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75f3a621e232421aa3a13213db373cb8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43747: connect: Connection refused (error 111)
I20260620 14:35:14.911178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19855
I20260620 14:35:14.915674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 19493
2026-06-20T14:35:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2001 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:14Z Disabled control of system clock
I20260620 14:35:14.942620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42071
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:15.023226 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:15.023448 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:15.023468 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:15.024560 20027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:15.024606 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:15.024618 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:15.024631 20027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:15.024643 20027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:15.025925 20027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:15.026187 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:15.026499 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:15.031781 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:15.031785 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:15.031777 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.031802 20027 server_base.cc:1034] running on GCE node
I20260620 14:35:15.032262 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:15.032545 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:15.033681 20027 hybrid_clock.cc:648] HybridClock initialized: now 1781966115033663 us; error 39 us; skew 500 ppm
I20260620 14:35:15.034875 20027 webserver.cc:460] Webserver started at http://127.0.98.62:36121/ using document root <none> and password file <none>
I20260620 14:35:15.035118 20027 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:15.035166 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:15.035254 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:15.036283 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "78a04d5126864ec6949842085671e2c4"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c46d2d7097a6f140736d2d236439960f"
server_key_iv: "851b0f1e4539d0f5035c8f1ece48241b"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.036576 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "78a04d5126864ec6949842085671e2c4"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c46d2d7097a6f140736d2d236439960f"
server_key_iv: "851b0f1e4539d0f5035c8f1ece48241b"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.037837 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:15.038501 20041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.038673 20027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:15.038733 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "78a04d5126864ec6949842085671e2c4"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c46d2d7097a6f140736d2d236439960f"
server_key_iv: "851b0f1e4539d0f5035c8f1ece48241b"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.038784 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:15.060475 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:15.060726 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:15.060832 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:15.064625 20027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42071
I20260620 14:35:15.064659 20093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42071 every 8 connection(s)
I20260620 14:35:15.065007 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:15.066049 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:15.066642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20027
I20260620 14:35:15.066730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:15.066936 392 external_mini_cluster.cc:1428] Setting key ee47075abd8cdb6a594707094e13bc25
I20260620 14:35:15.068884 20094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: Bootstrap starting.
I20260620 14:35:15.069545 20094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:15.069846 20094 log.cc:824] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:15.070569 20094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: No bootstrap required, opened a new log
I20260620 14:35:15.072440 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.067445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.0.98.62:42071)
0620 14:35:15.067624 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.067631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.067654 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:35:15.068071 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.068077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.068135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.068246 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:15.068489 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.069386 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.070477 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.070891 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.070976 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.071609 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:15.071612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:15.071612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:15.071757 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:15.071786 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:15.071795 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:15.071871 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.071969 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.072047 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260620 14:35:15.072358 20094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } }
I20260620 14:35:15.072538 20094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:15.072574 20094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a04d5126864ec6949842085671e2c4, State: Initialized, Role: FOLLOWER
I20260620 14:35:15.072690 20094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } }
I20260620 14:35:15.072762 20094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:15.072783 20094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:15.072851 20094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:15.073498 20094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } }
I20260620 14:35:15.073604 20094 leader_election.cc:302] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78a04d5126864ec6949842085671e2c4; no voters:
I20260620 14:35:15.073792 20094 leader_election.cc:288] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:15.073873 20099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:15.074007 20099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 78a04d5126864ec6949842085671e2c4, State: Running, Role: LEADER
I20260620 14:35:15.074091 20094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:15.074092 20099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } }
I20260620 14:35:15.074541 20100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78a04d5126864ec6949842085671e2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } } }
I20260620 14:35:15.074646 20100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:15.074545 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78a04d5126864ec6949842085671e2c4. Latest consensus state: current_term: 1 leader_uuid: "78a04d5126864ec6949842085671e2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a04d5126864ec6949842085671e2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42071 } } }
I20260620 14:35:15.074779 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:15.074893 20107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:15.075340 20107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:15.076754 20107 catalog_manager.cc:1347] Generated new cluster ID: 3be8db1919cd442985908d6fac562985
I20260620 14:35:15.076803 20107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:15.096616 20107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:15.097179 20107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:15.104887 20107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: Generated new TSK 0
I20260620 14:35:15.105098 20107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:15.110590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--builtin_ntp_servers=127.0.98.20:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:35:15.190510 20118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:15.190711 20118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:15.190732 20118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:15.191898 20118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:15.191939 20118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:15.191965 20118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:35:15.193336 20118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:15.193620 20118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:15.193935 20118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:15.199461 20124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:15.199525 20126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.199548 20118 server_base.cc:1034] running on GCE node
W20260620 14:35:15.199461 20123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.199910 20118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:15.200153 20118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:15.201295 20118 hybrid_clock.cc:648] HybridClock initialized: now 1781966115201278 us; error 35 us; skew 500 ppm
I20260620 14:35:15.202628 20118 webserver.cc:460] Webserver started at http://127.0.98.1:38541/ using document root <none> and password file <none>
I20260620 14:35:15.202900 20118 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:15.202951 20118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:15.203123 20118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:15.204185 20118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data/instance:
uuid: "3462e9a6f22541868241710231789b4d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "d85c4462d59e8c2b173a2ea38fd1e582"
server_key_iv: "1f0791c1f181cd3ba4776ac29b5799b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.204536 20118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3462e9a6f22541868241710231789b4d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "d85c4462d59e8c2b173a2ea38fd1e582"
server_key_iv: "1f0791c1f181cd3ba4776ac29b5799b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.205881 20118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:15.206595 20132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.206776 20118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:15.206847 20118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal
uuid: "3462e9a6f22541868241710231789b4d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "d85c4462d59e8c2b173a2ea38fd1e582"
server_key_iv: "1f0791c1f181cd3ba4776ac29b5799b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.206938 20118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:15.223932 20118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:15.224258 20118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:15.224421 20118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:15.224663 20118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:15.225004 20118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:15.225059 20118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.225104 20118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:15.225142 20118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.231186 20118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39015
I20260620 14:35:15.231179 20245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39015 every 8 connection(s)
I20260620 14:35:15.231586 20118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:35:15.235477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20118
I20260620 14:35:15.235595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:35:15.235819 392 external_mini_cluster.cc:1428] Setting key f2766e48ffb4a6013d100489a5fbcfa8
I20260620 14:35:15.236886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--builtin_ntp_servers=127.0.98.20:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:35:15.237505 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.232365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52967 (local address 127.0.98.62:42071)
0620 14:35:15.232451 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.232453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.232663 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:35:15.233212 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.233213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.233231 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.233297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:15.234063 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.234764 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.235871 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.236181 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.236253 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.237207 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:15.237208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:15.237208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:15.237213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:15.237229 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:15.237233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:15.237282 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.237374 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.237403 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:35:15.237728 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.232351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42071 (local address 127.0.98.1:52967)
0620 14:35:15.232536 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.232571 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.233092 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.233380 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.233384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.233391 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:15.233963 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.233968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.234864 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.234866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.235770 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.235772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.236272 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.236273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.236422 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.237049 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:15.237055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:15.237085 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:15.237108 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:15.237298 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:15.237303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:15.237306 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.237368 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.237499 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260620 14:35:15.238328 20246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42071
I20260620 14:35:15.238451 20246 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:15.238658 20246 heartbeater.cc:511] Master 127.0.98.62:42071 requested a full tablet report, sending...
I20260620 14:35:15.239176 20058 ts_manager.cc:194] Registered new tserver with Master: 3462e9a6f22541868241710231789b4d (127.0.98.1:39015)
I20260620 14:35:15.239799 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52967
W20260620 14:35:15.316895 20249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:15.317101 20249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:15.317121 20249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:15.318315 20249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:15.318358 20249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:15.318383 20249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:35:15.319725 20249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:15.319914 20249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:15.320214 20249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:15.324997 20255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.325044 20249 server_base.cc:1034] running on GCE node
W20260620 14:35:15.325016 20254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:15.325014 20257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.325407 20249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:15.325708 20249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:15.326884 20249 hybrid_clock.cc:648] HybridClock initialized: now 1781966115326878 us; error 34 us; skew 500 ppm
I20260620 14:35:15.328069 20249 webserver.cc:460] Webserver started at http://127.0.98.2:40893/ using document root <none> and password file <none>
I20260620 14:35:15.328315 20249 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:15.328361 20249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:15.328531 20249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:15.329625 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data/instance:
uuid: "67add8f321404f0bb217cdeaaed036f9"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c3857154aa1e7c21f05bcba1c054ed97"
server_key_iv: "7f3bc731b6a2a81c7578b64c395ff9ca"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.329985 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67add8f321404f0bb217cdeaaed036f9"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c3857154aa1e7c21f05bcba1c054ed97"
server_key_iv: "7f3bc731b6a2a81c7578b64c395ff9ca"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.331238 20249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:35:15.331936 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.332103 20249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:15.332211 20249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal
uuid: "67add8f321404f0bb217cdeaaed036f9"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "c3857154aa1e7c21f05bcba1c054ed97"
server_key_iv: "7f3bc731b6a2a81c7578b64c395ff9ca"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.332283 20249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:15.350767 20249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:15.351099 20249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:15.351265 20249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:15.351500 20249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:15.351845 20249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:15.351900 20249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.351943 20249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:15.351963 20249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.357498 20249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38557
I20260620 14:35:15.357581 20376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38557 every 8 connection(s)
I20260620 14:35:15.357915 20249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:35:15.362334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20249
I20260620 14:35:15.362452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:35:15.362679 392 external_mini_cluster.cc:1428] Setting key e9af5b7e8034560bda71e18bea7ec7bd
I20260620 14:35:15.363248 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.358703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44299 (local address 127.0.98.62:42071)
0620 14:35:15.358771 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.358773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.358954 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:35:15.359426 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.359427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.359437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.359495 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:15.360141 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.360647 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.361658 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.361840 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.361897 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.362947 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:15.362949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:15.362949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:15.362954 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:15.362972 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:15.362977 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:15.363032 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.363142 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.363165 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:35:15.363454 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.358630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42071 (local address 127.0.98.2:44299)
0620 14:35:15.358827 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.358866 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.359336 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.359551 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.359554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.359562 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:15.360039 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.360043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.360754 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.360755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.361567 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.361570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.362001 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.362002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.362181 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.362772 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:15.362772 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:15.362829 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:15.362858 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:15.363047 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:15.363053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:15.363056 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.363127 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.363218 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260620 14:35:15.363744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--builtin_ntp_servers=127.0.98.20:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:35:15.365197 20377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42071
I20260620 14:35:15.365290 20377 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:15.365510 20377 heartbeater.cc:511] Master 127.0.98.62:42071 requested a full tablet report, sending...
I20260620 14:35:15.365926 20058 ts_manager.cc:194] Registered new tserver with Master: 67add8f321404f0bb217cdeaaed036f9 (127.0.98.2:38557)
I20260620 14:35:15.366299 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44299
W20260620 14:35:15.445765 20380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:15.445996 20380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:15.446033 20380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:15.447189 20380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:15.447257 20380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:15.447316 20380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:35:15.448689 20380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:15.448952 20380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:15.449246 20380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:15.454182 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:15.454191 20388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.454475 20380 server_base.cc:1034] running on GCE node
W20260620 14:35:15.454191 20385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:15.454741 20380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:15.454986 20380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:15.456151 20380 hybrid_clock.cc:648] HybridClock initialized: now 1781966115456146 us; error 39 us; skew 500 ppm
I20260620 14:35:15.457463 20380 webserver.cc:460] Webserver started at http://127.0.98.3:35549/ using document root <none> and password file <none>
I20260620 14:35:15.457739 20380 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:15.457815 20380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:15.457967 20380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:15.459041 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data/instance:
uuid: "82c61144c5b04555a5458fa87519cb2d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "34fc111612063865c478c5bf97b9d3b1"
server_key_iv: "906fcbcb7483c615a0247a4ae128f37f"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.459411 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal/instance:
uuid: "82c61144c5b04555a5458fa87519cb2d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "34fc111612063865c478c5bf97b9d3b1"
server_key_iv: "906fcbcb7483c615a0247a4ae128f37f"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.460711 20380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:15.461390 20394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.461561 20380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:15.461652 20380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal
uuid: "82c61144c5b04555a5458fa87519cb2d"
format_stamp: "Formatted at 2026-06-20 14:35:15 on dist-test-slave-tc2g"
server_key: "34fc111612063865c478c5bf97b9d3b1"
server_key_iv: "906fcbcb7483c615a0247a4ae128f37f"
server_key_version: "encryptionkey@0"
I20260620 14:35:15.461735 20380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:15.473836 20380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:15.474164 20380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:15.474327 20380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:15.474581 20380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:35:15.474963 20380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:35:15.475010 20380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.475044 20380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:35:15.475077 20380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:15.482482 20380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44239
I20260620 14:35:15.482606 20507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44239 every 8 connection(s)
I20260620 14:35:15.482936 20380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:35:15.488596 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.483719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49743 (local address 127.0.98.62:42071)
0620 14:35:15.483768 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.483770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.484055 (+ 285us) server_negotiation.cc:407] Connection header received
0620 14:35:15.484678 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.484679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.484690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.484771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:15.485589 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.486103 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.487117 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.487319 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.487403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.488299 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:15.488300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:15.488301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:15.488306 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:15.488328 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:15.488332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:15.488393 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.488488 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.488517 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:15.488823 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.483715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42071 (local address 127.0.98.3:49743)
0620 14:35:15.483928 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.483972 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.484570 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.484864 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.484872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.484880 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:15.485482 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.485489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.486194 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.486195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.487005 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.487008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.487414 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.487415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.487559 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.488150 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:35:15.488150 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:15.488178 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:35:15.488201 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:15.488393 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:15.488399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:15.488402 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.488492 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.488595 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260620 14:35:15.489223 20508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42071
I20260620 14:35:15.489334 20508 heartbeater.cc:464] Registering TS with master...
I20260620 14:35:15.489568 20508 heartbeater.cc:511] Master 127.0.98.62:42071 requested a full tablet report, sending...
I20260620 14:35:15.490005 20058 ts_manager.cc:194] Registered new tserver with Master: 82c61144c5b04555a5458fa87519cb2d (127.0.98.3:44239)
I20260620 14:35:15.490134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20380
I20260620 14:35:15.490238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:35:15.490473 392 external_mini_cluster.cc:1428] Setting key 1ed63b3c382c124fee52ef95bd93f99b
I20260620 14:35:15.490463 20058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49743
I20260620 14:35:15.491394 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:35:15.496613 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.492981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.0.98.62:42071)
0620 14:35:15.493023 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.493025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.493192 (+ 167us) server_negotiation.cc:407] Connection header received
0620 14:35:15.493279 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.493280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.493290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.493339 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:15.493581 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.494306 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.495451 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.495743 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.495822 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.496396 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:15.496397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:15.496397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:15.496401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:15.496414 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:15.496417 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:15.496462 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.496498 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.496516 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:15.498208 20058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:35:15.499722 20058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:35:15.505952 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.502095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.0.98.3:44239)
0620 14:35:15.502282 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.502287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.502297 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:15.502337 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.502339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.502354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.502444 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:15.502704 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.503430 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.504854 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.505563 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505633 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.505730 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.505827 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.505872 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260620 14:35:15.505965 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.501843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44239 (local address 127.0.0.1:32772)
0620 14:35:15.502023 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.502063 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.502126 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.502458 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.502461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.502463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:15.502525 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.502528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.503520 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.503521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.504765 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.504768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.505647 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.505731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.505747 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.505816 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.505882 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":40,"thread_start_us":68,"threads_started":1}
I20260620 14:35:15.506083 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.501848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39015 (local address 127.0.0.1:51228)
0620 14:35:15.502212 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.502221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.502276 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.502540 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.502541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.502543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:15.502579 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.502580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.503904 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.503905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.504988 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.504990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.505872 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.505925 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.505928 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.505976 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.506012 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":32,"thread_start_us":104,"threads_started":1}
I20260620 14:35:15.506307 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.502128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.98.1:39015)
0620 14:35:15.502321 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.502325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.502334 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:15.502370 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.502372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.502392 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.502478 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:15.502656 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.503813 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505071 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.505685 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505752 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.505840 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.506185 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.506232 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260620 14:35:15.506702 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.501870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38557 (local address 127.0.0.1:47416)
0620 14:35:15.502508 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.502517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.502581 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.502881 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.502882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.502884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:15.502918 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.502919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.504085 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.504087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.505011 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.505013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.506476 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.506477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.506536 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.506541 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.506594 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.506632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260620 14:35:15.506783 20427 tablet_service.cc:1467] Processing CreateTablet for tablet 009ff67af13e4405bf495418ab27f9e6 (DEFAULT_TABLE table=test-table [id=8d31f5603737406587fe8dff8bf6c62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:15.506935 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.502309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.98.2:38557)
0620 14:35:15.502666 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.502670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.502680 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:15.502727 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.502729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.502743 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.502814 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:15.502991 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.504005 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505097 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.505840 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.505901 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.505979 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.506810 (+ 831us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.506856 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":103,"threads_started":1}
I20260620 14:35:15.506947 20180 tablet_service.cc:1467] Processing CreateTablet for tablet 009ff67af13e4405bf495418ab27f9e6 (DEFAULT_TABLE table=test-table [id=8d31f5603737406587fe8dff8bf6c62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:15.507133 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009ff67af13e4405bf495418ab27f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:15.507292 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009ff67af13e4405bf495418ab27f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:15.507682 20311 tablet_service.cc:1467] Processing CreateTablet for tablet 009ff67af13e4405bf495418ab27f9e6 (DEFAULT_TABLE table=test-table [id=8d31f5603737406587fe8dff8bf6c62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:35:15.508008 20311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009ff67af13e4405bf495418ab27f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:15.510406 20522 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: Bootstrap starting.
I20260620 14:35:15.510815 20523 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Bootstrap starting.
I20260620 14:35:15.511145 20524 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Bootstrap starting.
I20260620 14:35:15.511277 20522 tablet_bootstrap.cc:654] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:15.511518 20523 tablet_bootstrap.cc:654] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:15.511631 20522 log.cc:824] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:15.511791 20524 tablet_bootstrap.cc:654] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:15.511821 20523 log.cc:824] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:15.512143 20524 log.cc:824] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:15.512394 20522 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: No bootstrap required, opened a new log
I20260620 14:35:15.512472 20522 ts_tablet_manager.cc:1397] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:15.512831 20524 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: No bootstrap required, opened a new log
I20260620 14:35:15.512909 20524 ts_tablet_manager.cc:1397] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:35:15.515178 20522 raft_consensus.cc:348] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.515350 20522 raft_consensus.cc:374] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:15.515391 20522 raft_consensus.cc:729] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3462e9a6f22541868241710231789b4d, State: Initialized, Role: FOLLOWER
I20260620 14:35:15.515511 20522 consensus_queue.cc:260] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.515671 20524 raft_consensus.cc:348] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.515784 20522 ts_tablet_manager.cc:1428] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:15.515827 20524 raft_consensus.cc:374] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:15.515821 20246 heartbeater.cc:503] Master 127.0.98.62:42071 was elected leader, sending a full tablet report...
I20260620 14:35:15.515872 20524 raft_consensus.cc:729] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67add8f321404f0bb217cdeaaed036f9, State: Initialized, Role: FOLLOWER
I20260620 14:35:15.515991 20524 consensus_queue.cc:260] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.516185 20523 tablet_bootstrap.cc:492] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: No bootstrap required, opened a new log
I20260620 14:35:15.516255 20523 ts_tablet_manager.cc:1397] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 14:35:15.516278 20377 heartbeater.cc:503] Master 127.0.98.62:42071 was elected leader, sending a full tablet report...
I20260620 14:35:15.516930 20524 ts_tablet_manager.cc:1428] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:35:15.518430 20523 raft_consensus.cc:348] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.518544 20523 raft_consensus.cc:374] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:15.518564 20523 raft_consensus.cc:729] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c61144c5b04555a5458fa87519cb2d, State: Initialized, Role: FOLLOWER
I20260620 14:35:15.518695 20523 consensus_queue.cc:260] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.518929 20523 ts_tablet_manager.cc:1428] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:15.518985 20508 heartbeater.cc:503] Master 127.0.98.62:42071 was elected leader, sending a full tablet report...
W20260620 14:35:15.608486 20378 tablet.cc:2369] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:35:15.629604 20528 raft_consensus.cc:482] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:35:15.629769 20528 raft_consensus.cc:504] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.630270 20528 leader_election.cc:288] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c61144c5b04555a5458fa87519cb2d (127.0.98.3:44239), 67add8f321404f0bb217cdeaaed036f9 (127.0.98.2:38557)
I20260620 14:35:15.634402 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.630798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42467 (local address 127.0.98.3:44239)
0620 14:35:15.630841 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.630843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.630850 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:35:15.630877 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.630878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.630890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.630936 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:15.631117 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.631828 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.633441 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.634001 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.634050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.634062 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.634301 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.634342 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:15.634394 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.630462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38557 (local address 127.0.98.1:50263)
0620 14:35:15.630660 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.630669 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.630713 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.630898 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.630899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.630900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:15.630982 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.630985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.631937 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.631937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.633306 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.633310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.634136 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.634137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.634205 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.634209 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.634259 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.634319 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":61,"thread_start_us":108,"threads_started":1}
I20260620 14:35:15.634615 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.630428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44239 (local address 127.0.98.1:42467)
0620 14:35:15.630619 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.630630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:15.630692 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:15.630978 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:15.630979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:15.630980 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:35:15.631037 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.631039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.631936 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.631937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.633305 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:15.633310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.634127 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.634128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:15.634219 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.634222 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:35:15.634479 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 14:35:15.634522 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":53}
I20260620 14:35:15.634775 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.630518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50263 (local address 127.0.98.2:38557)
0620 14:35:15.630574 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.630576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.630708 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:35:15.630806 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.630807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.630819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.630892 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:35:15.631072 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.631853 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.633413 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.634001 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.634066 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.634075 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.634654 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.634704 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:35:15.635121 20331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ff67af13e4405bf495418ab27f9e6" candidate_uuid: "3462e9a6f22541868241710231789b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67add8f321404f0bb217cdeaaed036f9" is_pre_election: true
I20260620 14:35:15.635136 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ff67af13e4405bf495418ab27f9e6" candidate_uuid: "3462e9a6f22541868241710231789b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c61144c5b04555a5458fa87519cb2d" is_pre_election: true
I20260620 14:35:15.635313 20331 raft_consensus.cc:2454] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3462e9a6f22541868241710231789b4d in term 0.
I20260620 14:35:15.635313 20456 raft_consensus.cc:2454] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3462e9a6f22541868241710231789b4d in term 0.
I20260620 14:35:15.635566 20134 leader_election.cc:302] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3462e9a6f22541868241710231789b4d, 82c61144c5b04555a5458fa87519cb2d; no voters:
I20260620 14:35:15.635773 20528 raft_consensus.cc:2781] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:35:15.635836 20528 raft_consensus.cc:482] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:35:15.635869 20528 raft_consensus.cc:3037] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:15.636595 20528 raft_consensus.cc:504] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.636752 20528 leader_election.cc:288] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [CANDIDATE]: Term 1 election: Requested vote from peers 82c61144c5b04555a5458fa87519cb2d (127.0.98.3:44239), 67add8f321404f0bb217cdeaaed036f9 (127.0.98.2:38557)
I20260620 14:35:15.636921 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ff67af13e4405bf495418ab27f9e6" candidate_uuid: "3462e9a6f22541868241710231789b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c61144c5b04555a5458fa87519cb2d"
I20260620 14:35:15.637018 20456 raft_consensus.cc:3037] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:15.637336 20331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ff67af13e4405bf495418ab27f9e6" candidate_uuid: "3462e9a6f22541868241710231789b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67add8f321404f0bb217cdeaaed036f9"
I20260620 14:35:15.637451 20331 raft_consensus.cc:3037] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:15.637840 20456 raft_consensus.cc:2454] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3462e9a6f22541868241710231789b4d in term 1.
I20260620 14:35:15.638000 20134 leader_election.cc:302] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3462e9a6f22541868241710231789b4d, 82c61144c5b04555a5458fa87519cb2d; no voters:
I20260620 14:35:15.638103 20528 raft_consensus.cc:2781] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:15.638290 20528 raft_consensus.cc:686] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 1 LEADER]: Becoming Leader. State: Replica: 3462e9a6f22541868241710231789b4d, State: Running, Role: LEADER
I20260620 14:35:15.638397 20528 consensus_queue.cc:237] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260620 14:35:15.638515 20331 raft_consensus.cc:2454] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3462e9a6f22541868241710231789b4d in term 1.
I20260620 14:35:15.639405 20058 catalog_manager.cc:5495] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3462e9a6f22541868241710231789b4d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3462e9a6f22541868241710231789b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3462e9a6f22541868241710231789b4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } health_report { overall_health: UNKNOWN } } }
I20260620 14:35:15.649585 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.647390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.0.98.1:39015)
0620 14:35:15.647448 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:15.647450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:15.647507 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:35:15.647568 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:15.647568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:15.647584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:15.647637 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:15.647744 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.648197 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.648882 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:15.649054 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:15.649104 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:15.649225 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:15.649319 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:15.649482 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:15.649496 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 14:35:15.649514 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:35:15.651255 20331 raft_consensus.cc:1261] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 1 FOLLOWER]: Refusing update from remote peer 3462e9a6f22541868241710231789b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:15.651288 20456 raft_consensus.cc:1261] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 1 FOLLOWER]: Refusing update from remote peer 3462e9a6f22541868241710231789b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:15.651602 20528 consensus_queue.cc:1035] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67add8f321404f0bb217cdeaaed036f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:15.651739 20532 consensus_queue.cc:1035] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c61144c5b04555a5458fa87519cb2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:15.654572 20536 mvcc.cc:205] Tried to move back new op lower bound from 7298933209704992768 to 7298933209655545856. Current Snapshot: MvccSnapshot[applied={T|T < 7298933209704992768}]
I20260620 14:35:15.654758 20537 mvcc.cc:205] Tried to move back new op lower bound from 7298933209704992768 to 7298933209655545856. Current Snapshot: MvccSnapshot[applied={T|T < 7298933209704992768}]
I20260620 14:35:15.656105 20057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:15.656204 20057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:35:15.656994 20057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78a04d5126864ec6949842085671e2c4: Sending DeleteTablet for 3 replicas of tablet 009ff67af13e4405bf495418ab27f9e6
I20260620 14:35:15.657343 20427 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ff67af13e4405bf495418ab27f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:15 UTC) from {username='slave'} at 127.0.0.1:32772
I20260620 14:35:15.657377 20180 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ff67af13e4405bf495418ab27f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:15 UTC) from {username='slave'} at 127.0.0.1:51228
I20260620 14:35:15.657670 20311 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ff67af13e4405bf495418ab27f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:35:15 UTC) from {username='slave'} at 127.0.0.1:47416
I20260620 14:35:15.657740 20546 tablet_replica.cc:330] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d: stopping tablet replica
I20260620 14:35:15.657840 20546 raft_consensus.cc:2229] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:35:15.657922 20546 raft_consensus.cc:2258] T 009ff67af13e4405bf495418ab27f9e6 P 3462e9a6f22541868241710231789b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:15.658011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20118
I20260620 14:35:15.658051 20547 tablet_replica.cc:330] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: stopping tablet replica
I20260620 14:35:15.658215 20547 raft_consensus.cc:2229] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:15.658365 20547 raft_consensus.cc:2258] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:15.658664 20545 tablet_replica.cc:330] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: stopping tablet replica
I20260620 14:35:15.658762 20545 raft_consensus.cc:2229] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:35:15.658921 20545 raft_consensus.cc:2258] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:35:15.659018 20547 ts_tablet_manager.cc:1905] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:15.660907 20547 ts_tablet_manager.cc:1918] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:15.660969 20547 log.cc:1196] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-1/wal/wals/009ff67af13e4405bf495418ab27f9e6
I20260620 14:35:15.661211 20547 ts_tablet_manager.cc:1939] T 009ff67af13e4405bf495418ab27f9e6 P 67add8f321404f0bb217cdeaaed036f9: Deleting consensus metadata
I20260620 14:35:15.661674 20545 ts_tablet_manager.cc:1905] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:35:15.661670 20042 catalog_manager.cc:4841] TS 67add8f321404f0bb217cdeaaed036f9 (127.0.98.2:38557): tablet 009ff67af13e4405bf495418ab27f9e6 (table test-table [id=8d31f5603737406587fe8dff8bf6c62f]) successfully deleted
I20260620 14:35:15.662752 20545 ts_tablet_manager.cc:1918] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:35:15.662811 20545 log.cc:1196] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Deleting WAL directory at /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965876904925-392-0/minicluster-data/ts-2/wal/wals/009ff67af13e4405bf495418ab27f9e6
I20260620 14:35:15.663053 20545 ts_tablet_manager.cc:1939] T 009ff67af13e4405bf495418ab27f9e6 P 82c61144c5b04555a5458fa87519cb2d: Deleting consensus metadata
I20260620 14:35:15.663460 20043 catalog_manager.cc:4841] TS 82c61144c5b04555a5458fa87519cb2d (127.0.98.3:44239): tablet 009ff67af13e4405bf495418ab27f9e6 (table test-table [id=8d31f5603737406587fe8dff8bf6c62f]) successfully deleted
W20260620 14:35:15.663947 20044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39015 (error 108)
I20260620 14:35:15.664161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20249
I20260620 14:35:15.664800 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:15.664277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39015 (local address 127.0.0.1:51236)
0620 14:35:15.664723 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:15.664764 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
I20260620 14:35:15.668809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20380
W20260620 14:35:15.668807 20044 catalog_manager.cc:4584] TS 3462e9a6f22541868241710231789b4d (127.0.98.1:39015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 009ff67af13e4405bf495418ab27f9e6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39015: connect: Connection refused (error 111)
I20260620 14:35:15.672919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20027
2026-06-20T14:35:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (756 ms)
[----------] 6 tests from EncryptionPolicyTest (8173 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:15 dist-test-slave-tc2g krb5kdc[20551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:15 dist-test-slave-tc2g krb5kdc[20551](info): set up 2 sockets
Jun 20 14:35:15 dist-test-slave-tc2g krb5kdc[20551](info): commencing operation
krb5kdc: starting...
W20260620 14:35:16.701530 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:16 dist-test-slave-tc2g krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:16.812177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:16.891029 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:16.891263 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:16.891310 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:16.892474 20564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:16.892541 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:16.892565 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:16.892585 20564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:16.892613 20564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:16.894109 20564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:16.894385 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:16.894596 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:16.899495 20570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:16.899534 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:16.899600 20564 server_base.cc:1034] running on GCE node
W20260620 14:35:16.899469 20572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:16.899906 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:16.900185 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:16.901338 20564 hybrid_clock.cc:648] HybridClock initialized: now 1781966116901321 us; error 32 us; skew 500 ppm
Jun 20 14:35:16 dist-test-slave-tc2g krb5kdc[20551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:16.903679 20564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:16.904136 20564 webserver.cc:460] Webserver started at http://127.0.0.1:35457/ using document root <none> and password file <none>
I20260620 14:35:16.904357 20564 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:16.904402 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:16.904546 20564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:16.905683 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "7dcc5248bf124397b2143c70c1a71567"
format_stamp: "Formatted at 2026-06-20 14:35:16 on dist-test-slave-tc2g"
server_key: "07bf655887802717e979b2087532e9f8"
server_key_iv: "9e25b3769cb6083c726f3ca466091104"
server_key_version: "encryptionkey@0"
I20260620 14:35:16.906036 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dcc5248bf124397b2143c70c1a71567"
format_stamp: "Formatted at 2026-06-20 14:35:16 on dist-test-slave-tc2g"
server_key: "07bf655887802717e979b2087532e9f8"
server_key_iv: "9e25b3769cb6083c726f3ca466091104"
server_key_version: "encryptionkey@0"
I20260620 14:35:16.907308 20564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:16.907977 20579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:16.908144 20564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:16.908238 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "7dcc5248bf124397b2143c70c1a71567"
format_stamp: "Formatted at 2026-06-20 14:35:16 on dist-test-slave-tc2g"
server_key: "07bf655887802717e979b2087532e9f8"
server_key_iv: "9e25b3769cb6083c726f3ca466091104"
server_key_version: "encryptionkey@0"
I20260620 14:35:16.908322 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:16.923098 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:16.924810 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:16.924965 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:16.928996 20564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35049
I20260620 14:35:16.929028 20631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35049 every 8 connection(s)
I20260620 14:35:16.929399 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:16.930337 20632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:16.932538 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567: Bootstrap starting.
I20260620 14:35:16.933109 20632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:16.933393 20632 log.cc:824] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:16.934116 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567: No bootstrap required, opened a new log
I20260620 14:35:16.935664 20632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260620 14:35:16.935798 20632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:16.935820 20632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcc5248bf124397b2143c70c1a71567, State: Initialized, Role: FOLLOWER
I20260620 14:35:16.935990 20632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260620 14:35:16.936077 20632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:16.936129 20632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:16.936187 20632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:16.936429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20564
I20260620 14:35:16.936558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:16.936755 392 external_mini_cluster.cc:1428] Setting key 2d954f72adaa0d3dc35398225f18c3d2
I20260620 14:35:16.936849 20632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260620 14:35:16.936954 20632 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcc5248bf124397b2143c70c1a71567; no voters:
I20260620 14:35:16.937103 20632 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:16.937146 20635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:16.937291 20635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [term 1 LEADER]: Becoming Leader. State: Replica: 7dcc5248bf124397b2143c70c1a71567, State: Running, Role: LEADER
I20260620 14:35:16.937384 20632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:16.937377 20635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260620 14:35:16.937747 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcc5248bf124397b2143c70c1a71567. Latest consensus state: current_term: 1 leader_uuid: "7dcc5248bf124397b2143c70c1a71567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260620 14:35:16.937840 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:16.937731 20638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcc5248bf124397b2143c70c1a71567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcc5248bf124397b2143c70c1a71567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260620 14:35:16.938021 20638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:16.938133 20641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:16.938887 20641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:16.940536 20641 catalog_manager.cc:1347] Generated new cluster ID: 17cc56e2b46a4e29bc86efe046eed15f
I20260620 14:35:16.940585 20641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:35:16 dist-test-slave-tc2g krb5kdc[20551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:35:16.949110 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:16.937203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53553 (local address 127.0.0.1:35049)
0620 14:35:16.937473 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:16.937482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:16.937771 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:35:16.938161 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:16.938170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:16.938229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:16.938350 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:16.939632 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:16.940968 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:16.942026 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:16.942666 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:16.942780 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:16.945568 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:16.945571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:16.945572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:16.945583 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:16.947432 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:16.947901 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:16.947904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:16.947904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:16.948010 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:16.948249 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:16.948250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:16.948250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:16.948528 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:16.948680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:16.948823 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:35:16.948918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260620 14:35:16.950021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:16.960458 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:16.951432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42803 (local address 127.0.0.1:35049)
0620 14:35:16.951463 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:16.951465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:16.953596 (+ 2131us) server_negotiation.cc:407] Connection header received
0620 14:35:16.953658 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:16.953659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:16.953668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:16.953724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:16.954308 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:16.955025 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:16.956039 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:16.956359 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:16.956418 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:16.957777 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:16.957779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:16.957779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:16.957785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:16.959279 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:16.959692 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:16.959693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:16.959693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:16.959745 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:16.959977 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:16.959978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:16.959978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:16.960085 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:16.960188 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:16.960317 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:35:16.960373 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:35:16.965961 20641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:16.966461 20641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:16.968420 20641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dcc5248bf124397b2143c70c1a71567: Generated new TSK 0
I20260620 14:35:16.968603 20641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:16.974310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20564
W20260620 14:35:16.978783 20549 connection.cc:535] client connection to 127.0.0.1:35049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35049): Connection reset by peer (error 104)
2026-06-20T14:35:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:17 dist-test-slave-tc2g krb5kdc[20663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:17 dist-test-slave-tc2g krb5kdc[20663](info): set up 2 sockets
Jun 20 14:35:17 dist-test-slave-tc2g krb5kdc[20663](info): commencing operation
krb5kdc: starting...
W20260620 14:35:18.021407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:18 dist-test-slave-tc2g krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:18.134260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:18.213040 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:18.213302 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:18.213342 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:18.214550 20676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:18.214622 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:18.214645 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:18.214668 20676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:18.214697 20676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:18.216169 20676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:18.216452 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:18.216675 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:18.221419 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:18.221401 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:18.221470 20676 server_base.cc:1034] running on GCE node
W20260620 14:35:18.221411 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:18.221830 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:18.222139 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:18.223299 20676 hybrid_clock.cc:648] HybridClock initialized: now 1781966118223283 us; error 31 us; skew 500 ppm
Jun 20 14:35:18 dist-test-slave-tc2g krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:18.230716 20676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:18.231154 20676 webserver.cc:460] Webserver started at http://127.0.0.1:44867/ using document root <none> and password file <none>
I20260620 14:35:18.231393 20676 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:18.231462 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:18.231606 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:18.232677 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "4310ec2f02104648bbbb719ccadbcd85"
format_stamp: "Formatted at 2026-06-20 14:35:18 on dist-test-slave-tc2g"
server_key: "39c147f95c6d27fe7b0ce67055775443"
server_key_iv: "20c3e9e191215739c44caf4e3614e8bb"
server_key_version: "encryptionkey@0"
I20260620 14:35:18.233052 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "4310ec2f02104648bbbb719ccadbcd85"
format_stamp: "Formatted at 2026-06-20 14:35:18 on dist-test-slave-tc2g"
server_key: "39c147f95c6d27fe7b0ce67055775443"
server_key_iv: "20c3e9e191215739c44caf4e3614e8bb"
server_key_version: "encryptionkey@0"
I20260620 14:35:18.234396 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:18.235098 20691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:18.235263 20676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:18.235352 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "4310ec2f02104648bbbb719ccadbcd85"
format_stamp: "Formatted at 2026-06-20 14:35:18 on dist-test-slave-tc2g"
server_key: "39c147f95c6d27fe7b0ce67055775443"
server_key_iv: "20c3e9e191215739c44caf4e3614e8bb"
server_key_version: "encryptionkey@0"
I20260620 14:35:18.235428 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:18.244623 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:18.246351 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:18.246500 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:18.250401 20676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41943
I20260620 14:35:18.250429 20743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41943 every 8 connection(s)
I20260620 14:35:18.250752 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:18.251562 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:18.253844 20744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85: Bootstrap starting.
I20260620 14:35:18.254393 20744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:18.254669 20744 log.cc:824] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:18.255321 20744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85: No bootstrap required, opened a new log
I20260620 14:35:18.256848 20744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260620 14:35:18.256976 20744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:18.257033 20744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4310ec2f02104648bbbb719ccadbcd85, State: Initialized, Role: FOLLOWER
I20260620 14:35:18.257179 20744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260620 14:35:18.257242 20744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:18.257292 20744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:18.257342 20744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:18.257874 20744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260620 14:35:18.257992 20744 leader_election.cc:302] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4310ec2f02104648bbbb719ccadbcd85; no voters:
I20260620 14:35:18.258170 20744 leader_election.cc:288] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:18.258203 20747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:18.258375 20747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [term 1 LEADER]: Becoming Leader. State: Replica: 4310ec2f02104648bbbb719ccadbcd85, State: Running, Role: LEADER
I20260620 14:35:18.258422 20744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:18.258490 20747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260620 14:35:18.258862 20748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4310ec2f02104648bbbb719ccadbcd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260620 14:35:18.259002 20748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:18.259124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20676
I20260620 14:35:18.259250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:18.259461 392 external_mini_cluster.cc:1428] Setting key 13eb6dd376470dd45126cc5a7f5d7e69
I20260620 14:35:18.259445 20751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:18.258867 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4310ec2f02104648bbbb719ccadbcd85. Latest consensus state: current_term: 1 leader_uuid: "4310ec2f02104648bbbb719ccadbcd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310ec2f02104648bbbb719ccadbcd85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260620 14:35:18.259650 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:18.259969 20751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:18.261544 20751 catalog_manager.cc:1347] Generated new cluster ID: ae3e1840ea314de7b894e1668fc70763
I20260620 14:35:18.261597 20751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:18.264758 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:18.259899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42099 (local address 127.0.0.1:41943)
0620 14:35:18.260144 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:18.260155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:18.260276 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:35:18.260585 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:18.260594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:18.260656 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:18.260795 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:18.261062 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:18.262096 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:18.262917 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:18.263332 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:18.263446 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:18.264143 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:18.264146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:18.264147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:18.264328 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:18.264357 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:18.264368 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:18.264457 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:18.264535 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:35:18.264564 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260620 14:35:18.265663 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:18.271623 20751 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:18.272300 20751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:18.274889 20751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4310ec2f02104648bbbb719ccadbcd85: Generated new TSK 0
I20260620 14:35:18.275023 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:18.266956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36249 (local address 127.0.0.1:41943)
0620 14:35:18.266991 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:18.266993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:18.271367 (+ 4374us) server_negotiation.cc:407] Connection header received
0620 14:35:18.271394 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:18.271394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:18.271416 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:18.271483 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:18.272452 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:18.272932 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:18.273722 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:18.273936 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:18.274019 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:18.274693 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:18.274694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:18.274695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:18.274699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:18.274715 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:18.274720 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:18.274793 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:18.274937 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:35:18.274952 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:35:18.275233 20751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:18.282547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20676
W20260620 14:35:18.286924 20661 connection.cc:535] client connection to 127.0.0.1:41943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41943): Connection reset by peer (error 104)
2026-06-20T14:35:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:18 dist-test-slave-tc2g krb5kdc[20775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:18 dist-test-slave-tc2g krb5kdc[20775](info): set up 2 sockets
Jun 20 14:35:18 dist-test-slave-tc2g krb5kdc[20775](info): commencing operation
krb5kdc: starting...
W20260620 14:35:19.317571 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:19 dist-test-slave-tc2g krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:19.431562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:19.514966 20788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:19.515158 20788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:19.515182 20788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:19.516225 20788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:19.516256 20788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:19.516269 20788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:19.516283 20788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:19.516294 20788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:19.517694 20788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:19.517881 20788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:19.518044 20788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:19.522776 20793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:19.522823 20794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:19.522784 20796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:19.523100 20788 server_base.cc:1034] running on GCE node
I20260620 14:35:19.523344 20788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:19.523623 20788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:19.524801 20788 hybrid_clock.cc:648] HybridClock initialized: now 1781966119524798 us; error 40 us; skew 500 ppm
Jun 20 14:35:19 dist-test-slave-tc2g krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:19.527242 20788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:19.527657 20788 webserver.cc:460] Webserver started at http://127.0.0.1:36627/ using document root <none> and password file <none>
I20260620 14:35:19.527901 20788 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:19.527971 20788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:19.528112 20788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:19.529136 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "94df9fab53814780b4bd6477fb7114ac"
format_stamp: "Formatted at 2026-06-20 14:35:19 on dist-test-slave-tc2g"
server_key: "1954a9e876b131644670b704c532c03d"
server_key_iv: "4dd8fbe451957733fd5f6fe9e57d8793"
server_key_version: "encryptionkey@0"
I20260620 14:35:19.529536 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "94df9fab53814780b4bd6477fb7114ac"
format_stamp: "Formatted at 2026-06-20 14:35:19 on dist-test-slave-tc2g"
server_key: "1954a9e876b131644670b704c532c03d"
server_key_iv: "4dd8fbe451957733fd5f6fe9e57d8793"
server_key_version: "encryptionkey@0"
I20260620 14:35:19.530807 20788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:19.531457 20803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:19.531603 20788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:19.531699 20788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "94df9fab53814780b4bd6477fb7114ac"
format_stamp: "Formatted at 2026-06-20 14:35:19 on dist-test-slave-tc2g"
server_key: "1954a9e876b131644670b704c532c03d"
server_key_iv: "4dd8fbe451957733fd5f6fe9e57d8793"
server_key_version: "encryptionkey@0"
I20260620 14:35:19.531785 20788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:19.534957 20788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:19.536727 20788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:19.536886 20788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:19.540550 20788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37471
I20260620 14:35:19.540580 20855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37471 every 8 connection(s)
I20260620 14:35:19.540891 20788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:19.541910 20856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:19.544036 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac: Bootstrap starting.
I20260620 14:35:19.544689 20856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:19.544965 20856 log.cc:824] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:19.545635 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac: No bootstrap required, opened a new log
I20260620 14:35:19.546133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20788
I20260620 14:35:19.546272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:19.546476 392 external_mini_cluster.cc:1428] Setting key 337e83c25c9b1b4e6c5a9d2eef18ea17
I20260620 14:35:19.547705 20856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } }
I20260620 14:35:19.547894 20856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:19.547937 20856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94df9fab53814780b4bd6477fb7114ac, State: Initialized, Role: FOLLOWER
I20260620 14:35:19.548084 20856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } }
I20260620 14:35:19.548174 20856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:19.548211 20856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:19.548311 20856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:19.549067 20856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } }
I20260620 14:35:19.549191 20856 leader_election.cc:302] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94df9fab53814780b4bd6477fb7114ac; no voters:
I20260620 14:35:19.549225 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:19.546979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53291 (local address 127.0.0.1:37471)
0620 14:35:19.547144 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:19.547151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:19.547174 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:35:19.547865 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:19.547886 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:19.547935 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:19.548098 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:19.548740 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:19.548743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:19.548743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:19.548752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:19.548787 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:19.548797 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:19.548851 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:19.548940 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:35:19.548991 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 14:35:19.549401 20856 leader_election.cc:288] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:19.549450 20861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:19.549571 20861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [term 1 LEADER]: Becoming Leader. State: Replica: 94df9fab53814780b4bd6477fb7114ac, State: Running, Role: LEADER
I20260620 14:35:19.549683 20856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:19.549649 20861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } }
I20260620 14:35:19.549998 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94df9fab53814780b4bd6477fb7114ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } } }
I20260620 14:35:19.550104 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:19.550239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:19.550300 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94df9fab53814780b4bd6477fb7114ac. Latest consensus state: current_term: 1 leader_uuid: "94df9fab53814780b4bd6477fb7114ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94df9fab53814780b4bd6477fb7114ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37471 } } }
I20260620 14:35:19.550365 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:19.550477 20868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:19.550951 20868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:19.552795 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:19.551803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51753 (local address 127.0.0.1:37471)
0620 14:35:19.551840 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:19.551842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:19.551953 (+ 111us) server_negotiation.cc:407] Connection header received
0620 14:35:19.552018 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:19.552018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:19.552022 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:19.552062 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:19.552569 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:19.552570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:19.552570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:19.552573 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:19.552584 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:19.552588 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:19.552627 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:19.552686 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:35:19.552696 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:19.554423 20868 catalog_manager.cc:1347] Generated new cluster ID: 1794b1298d8f4bd29cb65604baa3ff08
I20260620 14:35:19.554486 20868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:19.560989 20868 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:19.561146 20868 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:19.576797 20868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94df9fab53814780b4bd6477fb7114ac: Generated new TSK 0
I20260620 14:35:19.577006 20868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:19.584021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20788
2026-06-20T14:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:19 dist-test-slave-tc2g krb5kdc[20887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:19 dist-test-slave-tc2g krb5kdc[20887](info): set up 2 sockets
Jun 20 14:35:19 dist-test-slave-tc2g krb5kdc[20887](info): commencing operation
krb5kdc: starting...
W20260620 14:35:20.629323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:20.745901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:20.824694 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:20.824932 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:20.824971 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:20.826193 20900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:20.826272 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:20.826298 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:20.826330 20900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:20.826355 20900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:20.827787 20900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:20.828073 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:20.828302 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:20.832984 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:20.832962 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:20.833069 20900 server_base.cc:1034] running on GCE node
W20260620 14:35:20.832974 20908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:20.833392 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:20.833745 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:20.834908 20900 hybrid_clock.cc:648] HybridClock initialized: now 1781966120834891 us; error 38 us; skew 500 ppm
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:20.837253 20900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:20.837754 20900 webserver.cc:460] Webserver started at http://127.0.0.1:35401/ using document root <none> and password file <none>
I20260620 14:35:20.837972 20900 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:20.838040 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:20.838193 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:20.839228 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "388e8c76d7da4f1095982bb40b666fce"
format_stamp: "Formatted at 2026-06-20 14:35:20 on dist-test-slave-tc2g"
server_key: "deeede870967534e198741fac44c077c"
server_key_iv: "6832484ad36e08c6e61de76ca4d4f0b9"
server_key_version: "encryptionkey@0"
I20260620 14:35:20.839601 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "388e8c76d7da4f1095982bb40b666fce"
format_stamp: "Formatted at 2026-06-20 14:35:20 on dist-test-slave-tc2g"
server_key: "deeede870967534e198741fac44c077c"
server_key_iv: "6832484ad36e08c6e61de76ca4d4f0b9"
server_key_version: "encryptionkey@0"
I20260620 14:35:20.840874 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:20.841619 20915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:20.841799 20900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:20.841895 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "388e8c76d7da4f1095982bb40b666fce"
format_stamp: "Formatted at 2026-06-20 14:35:20 on dist-test-slave-tc2g"
server_key: "deeede870967534e198741fac44c077c"
server_key_iv: "6832484ad36e08c6e61de76ca4d4f0b9"
server_key_version: "encryptionkey@0"
I20260620 14:35:20.841984 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:20.854254 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:20.855952 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:20.856108 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:20.860167 20900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36281
I20260620 14:35:20.860199 20967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36281 every 8 connection(s)
I20260620 14:35:20.860553 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:20.861366 20968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:20.863607 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce: Bootstrap starting.
I20260620 14:35:20.864174 20968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:20.864449 20968 log.cc:824] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:20.865092 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce: No bootstrap required, opened a new log
I20260620 14:35:20.866705 20968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 14:35:20.866850 20968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:20.866873 20968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388e8c76d7da4f1095982bb40b666fce, State: Initialized, Role: FOLLOWER
I20260620 14:35:20.867051 20968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 14:35:20.867113 20968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:20.867136 20968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:20.867208 20968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:20.867712 20968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 14:35:20.867820 20968 leader_election.cc:302] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 388e8c76d7da4f1095982bb40b666fce; no voters:
I20260620 14:35:20.867998 20968 leader_election.cc:288] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:20.868058 20971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:20.868230 20971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [term 1 LEADER]: Becoming Leader. State: Replica: 388e8c76d7da4f1095982bb40b666fce, State: Running, Role: LEADER
I20260620 14:35:20.868250 20968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:20.868358 20971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260620 14:35:20.868724 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "388e8c76d7da4f1095982bb40b666fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260620 14:35:20.868728 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 388e8c76d7da4f1095982bb40b666fce. Latest consensus state: current_term: 1 leader_uuid: "388e8c76d7da4f1095982bb40b666fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "388e8c76d7da4f1095982bb40b666fce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260620 14:35:20.868878 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:20.869086 20975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:20.869273 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:20.869686 20975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:20.870111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 20900
I20260620 14:35:20.870244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:20.870432 392 external_mini_cluster.cc:1428] Setting key f4c4f4ad234d796433ad6bd0ee662d56
I20260620 14:35:20.871300 20975 catalog_manager.cc:1347] Generated new cluster ID: 4235a730801143c68f8c5f15162b3e73
I20260620 14:35:20.871352 20975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:35:20.880390 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:20.871037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56303 (local address 127.0.0.1:36281)
0620 14:35:20.871456 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:20.871465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:20.871499 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:35:20.871589 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:20.871593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:20.871635 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:20.871765 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:20.872429 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:20.873357 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:20.874150 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:20.874590 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:20.874703 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:20.877041 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:20.877045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:20.877046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:20.877064 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:20.878828 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:20.879220 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:20.879224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:20.879224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:20.879333 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:20.879545 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:20.879546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:20.879546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:20.879787 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:20.879975 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:20.880042 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:35:20.880183 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":67,"threads_started":1}
I20260620 14:35:20.881351 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:20.886497 20975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:20.887158 20975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:20.889617 20975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 388e8c76d7da4f1095982bb40b666fce: Generated new TSK 0
I20260620 14:35:20.889825 20975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:20.891682 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:20.882711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44417 (local address 127.0.0.1:36281)
0620 14:35:20.882749 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:20.882751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:20.885665 (+ 2914us) server_negotiation.cc:407] Connection header received
0620 14:35:20.885741 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:20.885742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:20.885753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:20.885801 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:20.886438 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:20.886965 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:20.887760 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:20.887992 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:20.888066 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:20.889198 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:20.889199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:20.889199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:20.889206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:20.890581 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:20.890930 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:20.890931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:20.890931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:20.890982 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:20.891195 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:20.891196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:20.891196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:20.891306 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:20.891402 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:20.891510 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:35:20.891591 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:20.893810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 20900
2026-06-20T14:35:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20999](info): set up 2 sockets
Jun 20 14:35:20 dist-test-slave-tc2g krb5kdc[20999](info): commencing operation
krb5kdc: starting...
W20260620 14:35:21.929387 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:22 dist-test-slave-tc2g krb5kdc[20999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:22.042475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:22.121071 21012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:22.121316 21012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:22.121356 21012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:22.122504 21012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:22.122577 21012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:22.122601 21012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:22.122622 21012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:22.122651 21012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:22.124091 21012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:22.124364 21012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:22.124595 21012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:22.129233 21017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:22.129343 21012 server_base.cc:1034] running on GCE node
W20260620 14:35:22.129235 21020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:22.129307 21018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:22.129717 21012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:22.130034 21012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:22.131191 21012 hybrid_clock.cc:648] HybridClock initialized: now 1781966122131172 us; error 49 us; skew 500 ppm
Jun 20 14:35:22 dist-test-slave-tc2g krb5kdc[20999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:22.133569 21012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:22.134033 21012 webserver.cc:460] Webserver started at http://127.0.0.1:39977/ using document root <none> and password file <none>
I20260620 14:35:22.134274 21012 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:22.134341 21012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:22.134474 21012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:22.135465 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "82ee6f96f01f418d88190c4614565ab0"
format_stamp: "Formatted at 2026-06-20 14:35:22 on dist-test-slave-tc2g"
server_key: "a6cce8bd68aedc354cc6ab2cdbf7f025"
server_key_iv: "76edd194e559c38cfe223122de827b35"
server_key_version: "encryptionkey@0"
I20260620 14:35:22.135831 21012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "82ee6f96f01f418d88190c4614565ab0"
format_stamp: "Formatted at 2026-06-20 14:35:22 on dist-test-slave-tc2g"
server_key: "a6cce8bd68aedc354cc6ab2cdbf7f025"
server_key_iv: "76edd194e559c38cfe223122de827b35"
server_key_version: "encryptionkey@0"
I20260620 14:35:22.137094 21012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:22.137873 21027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:22.138042 21012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:22.138131 21012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "82ee6f96f01f418d88190c4614565ab0"
format_stamp: "Formatted at 2026-06-20 14:35:22 on dist-test-slave-tc2g"
server_key: "a6cce8bd68aedc354cc6ab2cdbf7f025"
server_key_iv: "76edd194e559c38cfe223122de827b35"
server_key_version: "encryptionkey@0"
I20260620 14:35:22.138216 21012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:22.153775 21012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:22.155417 21012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:22.155570 21012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:22.159648 21012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43059
I20260620 14:35:22.159722 21079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43059 every 8 connection(s)
I20260620 14:35:22.160065 21012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:22.160861 21080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:22.163084 21080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: Bootstrap starting.
I20260620 14:35:22.163586 21080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:22.163807 21080 log.cc:824] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:22.164404 21080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: No bootstrap required, opened a new log
I20260620 14:35:22.165985 21080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260620 14:35:22.166138 21080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:22.166160 21080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee6f96f01f418d88190c4614565ab0, State: Initialized, Role: FOLLOWER
I20260620 14:35:22.166301 21080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260620 14:35:22.166393 21080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:22.166425 21080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:22.166469 21080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:22.166505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21012
I20260620 14:35:22.166631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:22.166834 392 external_mini_cluster.cc:1428] Setting key 8ce6c2974284f61f66ec8106f1ddda0f
I20260620 14:35:22.167182 21080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260620 14:35:22.167388 21080 leader_election.cc:302] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82ee6f96f01f418d88190c4614565ab0; no voters:
I20260620 14:35:22.167563 21080 leader_election.cc:288] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:22.167608 21085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:22.167780 21085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 82ee6f96f01f418d88190c4614565ab0, State: Running, Role: LEADER
I20260620 14:35:22.167824 21080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:22.167918 21085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260620 14:35:22.170030 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82ee6f96f01f418d88190c4614565ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } } }
I20260620 14:35:22.170176 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82ee6f96f01f418d88190c4614565ab0. Latest consensus state: current_term: 1 leader_uuid: "82ee6f96f01f418d88190c4614565ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ee6f96f01f418d88190c4614565ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } } }
I20260620 14:35:22.170218 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:22.170220 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0 [sys.catalog]: This master's current role is: LEADER
W20260620 14:35:22.170684 21099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:22.170714 21099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:22.170779 21101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:22.170994 21101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:22.172484 21101 catalog_manager.cc:1347] Generated new cluster ID: f4129a0216f64d1b9b6420867586e54b
I20260620 14:35:22.172537 21101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:22.174264 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:22.167416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46819 (local address 127.0.0.1:43059)
0620 14:35:22.167586 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:22.167595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:22.167629 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:35:22.167745 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:22.167752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:22.167803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:22.167974 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:22.168780 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:22.170268 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:22.171388 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:22.171846 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:22.171977 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:22.173668 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:22.173671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:22.173672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:22.173851 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:22.173878 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:22.173888 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:22.173969 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:22.174042 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:35:22.174112 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260620 14:35:22.175217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:22.184310 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:22.177729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46441 (local address 127.0.0.1:43059)
0620 14:35:22.177771 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:22.177773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:22.181645 (+ 3872us) server_negotiation.cc:407] Connection header received
0620 14:35:22.181722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:22.181723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:22.181733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:22.181803 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:22.181952 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:22.182414 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:22.183109 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:22.183313 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:22.183400 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:22.184063 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:22.184064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:22.184064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:22.184071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:22.184084 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:22.184088 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:22.184141 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:22.184205 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:35:22.184229 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:22.184561 21101 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:22.185268 21101 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:22.192965 21101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82ee6f96f01f418d88190c4614565ab0: Generated new TSK 0
I20260620 14:35:22.193141 21101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:22.195151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21012
2026-06-20T14:35:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:22 dist-test-slave-tc2g krb5kdc[21111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:22 dist-test-slave-tc2g krb5kdc[21111](info): set up 2 sockets
Jun 20 14:35:22 dist-test-slave-tc2g krb5kdc[21111](info): commencing operation
krb5kdc: starting...
W20260620 14:35:23.225621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:23 dist-test-slave-tc2g krb5kdc[21111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:23.338460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:23.418251 21124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:23.418495 21124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:23.418534 21124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:23.419691 21124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:23.419762 21124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:23.419787 21124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:23.419807 21124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:23.419840 21124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:23.421348 21124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:23.421656 21124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:23.421895 21124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:23.426564 21132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:23.426594 21130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:23.426692 21124 server_base.cc:1034] running on GCE node
W20260620 14:35:23.426564 21129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:23.427073 21124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:23.427317 21124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:23.428453 21124 hybrid_clock.cc:648] HybridClock initialized: now 1781966123428440 us; error 29 us; skew 500 ppm
Jun 20 14:35:23 dist-test-slave-tc2g krb5kdc[21111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:23.433956 21124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:23.434415 21124 webserver.cc:460] Webserver started at http://127.0.0.1:33831/ using document root <none> and password file <none>
I20260620 14:35:23.434652 21124 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:23.434696 21124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:23.434864 21124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:23.435890 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "0e5ea7eb535943d1a2e3842814216577"
format_stamp: "Formatted at 2026-06-20 14:35:23 on dist-test-slave-tc2g"
server_key: "46bbd9757a50076a7a296a5e32cddd2e"
server_key_iv: "270b611c9a2e3913754d30fe92cec3dd"
server_key_version: "encryptionkey@0"
I20260620 14:35:23.436240 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e5ea7eb535943d1a2e3842814216577"
format_stamp: "Formatted at 2026-06-20 14:35:23 on dist-test-slave-tc2g"
server_key: "46bbd9757a50076a7a296a5e32cddd2e"
server_key_iv: "270b611c9a2e3913754d30fe92cec3dd"
server_key_version: "encryptionkey@0"
I20260620 14:35:23.437577 21124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:23.438305 21139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:23.438493 21124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:23.438580 21124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "0e5ea7eb535943d1a2e3842814216577"
format_stamp: "Formatted at 2026-06-20 14:35:23 on dist-test-slave-tc2g"
server_key: "46bbd9757a50076a7a296a5e32cddd2e"
server_key_iv: "270b611c9a2e3913754d30fe92cec3dd"
server_key_version: "encryptionkey@0"
I20260620 14:35:23.438653 21124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:23.441936 21124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:23.444198 21124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:23.444356 21124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:23.448324 21124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43159
I20260620 14:35:23.448354 21191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43159 every 8 connection(s)
I20260620 14:35:23.448668 21124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:23.449654 21192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:23.451742 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577: Bootstrap starting.
I20260620 14:35:23.452248 21192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:23.452478 21192 log.cc:824] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:23.452878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21124
I20260620 14:35:23.452999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:23.453194 392 external_mini_cluster.cc:1428] Setting key 6c91f35f507a2d405003407418e7f704
I20260620 14:35:23.453208 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577: No bootstrap required, opened a new log
I20260620 14:35:23.455030 21192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260620 14:35:23.455189 21192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:23.455284 21192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5ea7eb535943d1a2e3842814216577, State: Initialized, Role: FOLLOWER
I20260620 14:35:23.455446 21192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260620 14:35:23.455542 21192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:23.455586 21192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:23.455633 21192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:23.456218 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:23.453971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42197 (local address 127.0.0.1:43159)
0620 14:35:23.454157 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:23.454164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:23.454187 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:35:23.455018 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:23.455023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:23.455047 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:23.455187 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:23.455765 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:23.455768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:23.455768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:23.455772 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:23.455818 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:23.455828 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:23.455903 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:23.455970 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:35:23.456009 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260620 14:35:23.456698 21192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260620 14:35:23.456807 21192 leader_election.cc:302] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5ea7eb535943d1a2e3842814216577; no voters:
I20260620 14:35:23.456970 21192 leader_election.cc:288] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:23.457155 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:23.457163 21197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:23.457247 21192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:23.457306 21197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5ea7eb535943d1a2e3842814216577, State: Running, Role: LEADER
I20260620 14:35:23.457372 21197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260620 14:35:23.457728 21198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5ea7eb535943d1a2e3842814216577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } } }
I20260620 14:35:23.457731 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5ea7eb535943d1a2e3842814216577. Latest consensus state: current_term: 1 leader_uuid: "0e5ea7eb535943d1a2e3842814216577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ea7eb535943d1a2e3842814216577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } } }
I20260620 14:35:23.457818 21198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:23.457844 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:23.458003 21206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:23.458571 21206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:23.460238 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:23.458945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35715 (local address 127.0.0.1:43159)
0620 14:35:23.458978 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:23.458980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:23.459202 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:35:23.459323 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:23.459324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:23.459331 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:23.459368 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:23.459968 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:23.459969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:23.459969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:23.459973 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:23.459987 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:23.459992 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:23.460101 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:23.460163 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:35:23.460179 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:23.460531 21206 catalog_manager.cc:1347] Generated new cluster ID: 508010aa5d4b4c9b9bb7822618526896
I20260620 14:35:23.460621 21206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:23.475051 21206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:23.475286 21206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:23.481916 21206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e5ea7eb535943d1a2e3842814216577: Generated new TSK 0
I20260620 14:35:23.482101 21206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:23.488791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21124
2026-06-20T14:35:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:23 dist-test-slave-tc2g krb5kdc[21223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:23 dist-test-slave-tc2g krb5kdc[21223](info): set up 2 sockets
Jun 20 14:35:23 dist-test-slave-tc2g krb5kdc[21223](info): commencing operation
krb5kdc: starting...
W20260620 14:35:24.517410 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:24.631928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:24.713721 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:24.713925 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:24.713948 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:24.715037 21236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:24.715071 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:24.715085 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:24.715097 21236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:24.715109 21236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:24.716507 21236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:24.716701 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:24.716861 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:24.721566 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:24.721561 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:24.721561 21241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:24.721715 21236 server_base.cc:1034] running on GCE node
I20260620 14:35:24.721926 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:24.722199 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:24.723371 21236 hybrid_clock.cc:648] HybridClock initialized: now 1781966124723349 us; error 35 us; skew 500 ppm
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:24.726029 21236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:24.726507 21236 webserver.cc:460] Webserver started at http://127.0.0.1:41801/ using document root <none> and password file <none>
I20260620 14:35:24.726742 21236 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:24.726812 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:24.726964 21236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:24.728020 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "c48737ae0d3943d98ce95b5ede01420d"
format_stamp: "Formatted at 2026-06-20 14:35:24 on dist-test-slave-tc2g"
server_key: "e2f046f31f1f73f2dc7664ccb4113ea5"
server_key_iv: "3388e356c9f10e6c50d1c09dc2d82f9e"
server_key_version: "encryptionkey@0"
I20260620 14:35:24.728385 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "c48737ae0d3943d98ce95b5ede01420d"
format_stamp: "Formatted at 2026-06-20 14:35:24 on dist-test-slave-tc2g"
server_key: "e2f046f31f1f73f2dc7664ccb4113ea5"
server_key_iv: "3388e356c9f10e6c50d1c09dc2d82f9e"
server_key_version: "encryptionkey@0"
I20260620 14:35:24.729779 21236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:24.730489 21251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:24.730652 21236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:24.730762 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "c48737ae0d3943d98ce95b5ede01420d"
format_stamp: "Formatted at 2026-06-20 14:35:24 on dist-test-slave-tc2g"
server_key: "e2f046f31f1f73f2dc7664ccb4113ea5"
server_key_iv: "3388e356c9f10e6c50d1c09dc2d82f9e"
server_key_version: "encryptionkey@0"
I20260620 14:35:24.730839 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:24.767354 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:24.769284 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:24.769470 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:24.773195 21236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35649
I20260620 14:35:24.773252 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35649 every 8 connection(s)
I20260620 14:35:24.773628 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:24.774487 21304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:24.776746 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d: Bootstrap starting.
I20260620 14:35:24.777014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21236
I20260620 14:35:24.777154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:24.777359 392 external_mini_cluster.cc:1428] Setting key c8da6cd9353559d8f65c4ee69e3b148f
I20260620 14:35:24.777393 21304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:24.777745 21304 log.cc:824] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:24.778448 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d: No bootstrap required, opened a new log
I20260620 14:35:24.780269 21304 raft_consensus.cc:348] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } }
I20260620 14:35:24.780400 21304 raft_consensus.cc:374] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:24.780421 21304 raft_consensus.cc:729] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48737ae0d3943d98ce95b5ede01420d, State: Initialized, Role: FOLLOWER
I20260620 14:35:24.780553 21304 consensus_queue.cc:260] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } }
I20260620 14:35:24.780646 21304 raft_consensus.cc:388] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:24.780689 21304 raft_consensus.cc:482] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:24.780737 21304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:24.781255 21304 raft_consensus.cc:504] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } }
I20260620 14:35:24.781371 21304 leader_election.cc:302] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c48737ae0d3943d98ce95b5ede01420d; no voters:
I20260620 14:35:24.781549 21304 leader_election.cc:288] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:24.781574 21309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:24.781697 21309 raft_consensus.cc:686] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [term 1 LEADER]: Becoming Leader. State: Replica: c48737ae0d3943d98ce95b5ede01420d, State: Running, Role: LEADER
I20260620 14:35:24.781810 21304 sys_catalog.cc:564] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:24.781841 21309 consensus_queue.cc:237] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } }
I20260620 14:35:24.782198 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c48737ae0d3943d98ce95b5ede01420d. Latest consensus state: current_term: 1 leader_uuid: "c48737ae0d3943d98ce95b5ede01420d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } } }
I20260620 14:35:24.782337 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:24.782358 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c48737ae0d3943d98ce95b5ede01420d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48737ae0d3943d98ce95b5ede01420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35649 } } }
I20260620 14:35:24.782544 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:24.782609 21318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:24.783914 21318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:35:24.786022 21318 catalog_manager.cc:1347] Generated new cluster ID: 522917d4e6c64b14baf2d2145a155228
I20260620 14:35:24.786079 21318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:24.788213 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:24.777875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:56977 (local address 127.0.0.1:35649)
0620 14:35:24.778082 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:24.778092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:24.778147 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:35:24.778697 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:24.778704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:24.778763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:24.778918 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:24.779557 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:24.780519 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:24.781356 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:24.781962 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:24.782063 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:24.784767 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:24.784771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:24.784771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:24.784782 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:24.786660 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:24.787066 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:24.787069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:24.787070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:24.787157 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:24.787394 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:24.787395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:24.787395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:24.787622 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:24.787828 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:24.787922 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:35:24.788023 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260620 14:35:24.789134 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:24.796638 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:24.790502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:49701 (local address 127.0.0.1:35649)
0620 14:35:24.790535 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:24.790537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:24.790643 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:35:24.790714 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:24.790715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:24.790725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:24.790788 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:24.791296 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:24.791774 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:24.792468 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:24.792714 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:24.792805 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:24.794039 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:24.794040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:24.794040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:24.794048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:24.795451 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:24.795814 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:24.795815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:24.795816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:24.795872 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:24.796102 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:24.796102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:24.796103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:24.796213 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:24.796337 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:24.796460 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:35:24.796522 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:24.832923 21318 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:24.833722 21318 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:24.836660 21318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c48737ae0d3943d98ce95b5ede01420d: Generated new TSK 0
I20260620 14:35:24.836854 21318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:24.841688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21236
W20260620 14:35:24.846086 21221 connection.cc:535] client connection to 127.0.0.1:35649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35649): Connection reset by peer (error 104)
2026-06-20T14:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21335](info): set up 2 sockets
Jun 20 14:35:24 dist-test-slave-tc2g krb5kdc[21335](info): commencing operation
krb5kdc: starting...
W20260620 14:35:25.873593 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:25 dist-test-slave-tc2g krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:25.985183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:26.065215 21348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:26.065497 21348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:26.065527 21348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:26.066630 21348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:26.066666 21348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:26.066680 21348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:26.066694 21348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:26.066704 21348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:26.068065 21348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:26.068260 21348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:26.068423 21348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:26.072970 21354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:26.073040 21356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:26.072973 21353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:26.073257 21348 server_base.cc:1034] running on GCE node
I20260620 14:35:26.073592 21348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:26.073836 21348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:26.074998 21348 hybrid_clock.cc:648] HybridClock initialized: now 1781966126074992 us; error 36 us; skew 500 ppm
Jun 20 14:35:26 dist-test-slave-tc2g krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:26.077484 21348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:26.077931 21348 webserver.cc:460] Webserver started at http://127.0.0.1:46093/ using document root <none> and password file <none>
I20260620 14:35:26.078155 21348 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:26.078230 21348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:26.078384 21348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:26.079397 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "c891acdb75ea4c6b9843db9ceffc7dea"
format_stamp: "Formatted at 2026-06-20 14:35:26 on dist-test-slave-tc2g"
server_key: "38e1228afb1941b0eb596210d4fbe108"
server_key_iv: "29502f0b48257a90ae94208b37c7f9d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:26.079752 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "c891acdb75ea4c6b9843db9ceffc7dea"
format_stamp: "Formatted at 2026-06-20 14:35:26 on dist-test-slave-tc2g"
server_key: "38e1228afb1941b0eb596210d4fbe108"
server_key_iv: "29502f0b48257a90ae94208b37c7f9d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:26.081015 21348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:26.081890 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:26.082082 21348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:35:26.082180 21348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "c891acdb75ea4c6b9843db9ceffc7dea"
format_stamp: "Formatted at 2026-06-20 14:35:26 on dist-test-slave-tc2g"
server_key: "38e1228afb1941b0eb596210d4fbe108"
server_key_iv: "29502f0b48257a90ae94208b37c7f9d8"
server_key_version: "encryptionkey@0"
I20260620 14:35:26.082276 21348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:26.101584 21348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:26.103953 21348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:26.104121 21348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:26.108093 21348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45665
I20260620 14:35:26.108130 21415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45665 every 8 connection(s)
I20260620 14:35:26.108511 21348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:26.109339 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:26.109591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21348
I20260620 14:35:26.109710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:26.109913 392 external_mini_cluster.cc:1428] Setting key 12cb08a0d1336b9ac173483afed1cb22
I20260620 14:35:26.112469 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea: Bootstrap starting.
I20260620 14:35:26.113152 21416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:26.113524 21416 log.cc:824] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:26.114217 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea: No bootstrap required, opened a new log
I20260620 14:35:26.116257 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:26.110476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:51419 (local address 127.0.0.1:45665)
0620 14:35:26.110707 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:26.110717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:26.110758 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:35:26.111286 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:26.111295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:26.111360 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:26.111518 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:26.111719 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:26.113093 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:26.113989 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:26.114575 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:26.114691 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:26.115328 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:26.115333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:26.115333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:26.115539 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:26.115578 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:26.115592 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:26.115684 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:26.115773 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:35:26.115799 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260620 14:35:26.116281 21416 raft_consensus.cc:348] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260620 14:35:26.116439 21416 raft_consensus.cc:374] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:26.116478 21416 raft_consensus.cc:729] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c891acdb75ea4c6b9843db9ceffc7dea, State: Initialized, Role: FOLLOWER
I20260620 14:35:26.116590 21416 consensus_queue.cc:260] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260620 14:35:26.116671 21416 raft_consensus.cc:388] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:26.116700 21416 raft_consensus.cc:482] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:26.116739 21416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:26.117347 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:26.117473 21416 raft_consensus.cc:504] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260620 14:35:26.117583 21416 leader_election.cc:302] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c891acdb75ea4c6b9843db9ceffc7dea; no voters:
I20260620 14:35:26.117779 21416 leader_election.cc:288] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:26.117825 21421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:26.117986 21421 raft_consensus.cc:686] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [term 1 LEADER]: Becoming Leader. State: Replica: c891acdb75ea4c6b9843db9ceffc7dea, State: Running, Role: LEADER
I20260620 14:35:26.118019 21416 sys_catalog.cc:564] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:26.118166 21421 consensus_queue.cc:237] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260620 14:35:26.118501 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } } }
I20260620 14:35:26.118590 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:26.118862 21433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:26.119349 21433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:26.119776 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c891acdb75ea4c6b9843db9ceffc7dea. Latest consensus state: current_term: 1 leader_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c891acdb75ea4c6b9843db9ceffc7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } } }
I20260620 14:35:26.119904 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:26.121114 21433 catalog_manager.cc:1347] Generated new cluster ID: bd1d364b7e4f46feb75be6acdc88da58
I20260620 14:35:26.121170 21433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:26.122656 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:26.118930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:44277 (local address 127.0.0.1:45665)
0620 14:35:26.118968 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:26.118970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:26.119644 (+ 674us) server_negotiation.cc:407] Connection header received
0620 14:35:26.119747 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:26.119748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:26.119761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:26.119808 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:26.120011 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:26.120716 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:26.121532 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:26.121749 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:26.121832 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:26.122440 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:26.122442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:26.122442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:26.122446 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:26.122460 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:26.122464 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:26.122516 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:26.122577 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:35:26.122585 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:26.145831 21433 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:26.146378 21433 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:26.151207 21433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c891acdb75ea4c6b9843db9ceffc7dea: Generated new TSK 0
I20260620 14:35:26.151397 21433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:26.159229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21348
W20260620 14:35:26.163568 21333 connection.cc:535] client connection to 127.0.0.1:45665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45665): Connection reset by peer (error 104)
2026-06-20T14:35:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:26 dist-test-slave-tc2g krb5kdc[21447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:26 dist-test-slave-tc2g krb5kdc[21447](info): set up 2 sockets
Jun 20 14:35:26 dist-test-slave-tc2g krb5kdc[21447](info): commencing operation
krb5kdc: starting...
W20260620 14:35:27.181577 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:27 dist-test-slave-tc2g krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:27.294312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:27.373235 21460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:27.373478 21460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:27.373520 21460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:27.374724 21460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:27.374797 21460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:27.374821 21460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:27.374850 21460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:27.374881 21460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:27.376338 21460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:27.376595 21460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:27.376827 21460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:27.381430 21465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:27.381567 21466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:27.381744 21460 server_base.cc:1034] running on GCE node
W20260620 14:35:27.381786 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:27.382023 21460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:27.382304 21460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:27.383477 21460 hybrid_clock.cc:648] HybridClock initialized: now 1781966127383457 us; error 31 us; skew 500 ppm
Jun 20 14:35:27 dist-test-slave-tc2g krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:27.385911 21460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:27.386310 21460 webserver.cc:460] Webserver started at http://127.0.0.1:41207/ using document root <none> and password file <none>
I20260620 14:35:27.386545 21460 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:27.386611 21460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:27.386790 21460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:27.387812 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "420302ca839147a2bdfacea357dbf8a5"
format_stamp: "Formatted at 2026-06-20 14:35:27 on dist-test-slave-tc2g"
server_key: "ecbf0039f61281587680dbb729bc05cb"
server_key_iv: "a68299a351a8c092c5c50d985d98bebf"
server_key_version: "encryptionkey@0"
I20260620 14:35:27.388165 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "420302ca839147a2bdfacea357dbf8a5"
format_stamp: "Formatted at 2026-06-20 14:35:27 on dist-test-slave-tc2g"
server_key: "ecbf0039f61281587680dbb729bc05cb"
server_key_iv: "a68299a351a8c092c5c50d985d98bebf"
server_key_version: "encryptionkey@0"
I20260620 14:35:27.389559 21460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:27.390511 21475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:27.390695 21460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:27.390816 21460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "420302ca839147a2bdfacea357dbf8a5"
format_stamp: "Formatted at 2026-06-20 14:35:27 on dist-test-slave-tc2g"
server_key: "ecbf0039f61281587680dbb729bc05cb"
server_key_iv: "a68299a351a8c092c5c50d985d98bebf"
server_key_version: "encryptionkey@0"
I20260620 14:35:27.390890 21460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:27.394570 21460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:27.397367 21460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:27.397558 21460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:27.401849 21460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36223
I20260620 14:35:27.401877 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36223 every 8 connection(s)
I20260620 14:35:27.402197 21460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:27.403131 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:27.405844 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: Bootstrap starting.
I20260620 14:35:27.406355 21528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:27.406579 21528 log.cc:824] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:27.407198 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: No bootstrap required, opened a new log
I20260620 14:35:27.408818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21460
I20260620 14:35:27.408730 21528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260620 14:35:27.408878 21528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:27.408938 21528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420302ca839147a2bdfacea357dbf8a5, State: Initialized, Role: FOLLOWER
I20260620 14:35:27.408947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:27.409099 21528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260620 14:35:27.409160 21528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:27.409183 392 external_mini_cluster.cc:1428] Setting key c6952a13dc38ab725caaf19d03962fe1
I20260620 14:35:27.409184 21528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:27.409252 21528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:27.410410 21528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260620 14:35:27.410522 21528 leader_election.cc:302] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 420302ca839147a2bdfacea357dbf8a5; no voters:
I20260620 14:35:27.410676 21528 leader_election.cc:288] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:27.410825 21533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:27.410917 21528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:27.411016 21533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [term 1 LEADER]: Becoming Leader. State: Replica: 420302ca839147a2bdfacea357dbf8a5, State: Running, Role: LEADER
I20260620 14:35:27.411180 21533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260620 14:35:27.411645 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:27.409743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:50607 (local address 127.0.0.1:36223)
0620 14:35:27.409946 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:27.409958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:27.409982 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:35:27.410071 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:27.410077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:27.410100 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:27.410209 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:27.411004 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:27.411008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:27.411008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:27.411014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:27.411057 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:27.411073 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:27.411165 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:27.411400 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:35:27.411444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":123,"threads_started":1}
I20260620 14:35:27.412626 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:35:27.412976 21548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:27.413039 21548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:27.412352 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "420302ca839147a2bdfacea357dbf8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } } }
I20260620 14:35:27.413996 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:27.412541 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 420302ca839147a2bdfacea357dbf8a5. Latest consensus state: current_term: 1 leader_uuid: "420302ca839147a2bdfacea357dbf8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420302ca839147a2bdfacea357dbf8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } } }
I20260620 14:35:27.414149 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:27.414309 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:27.414496 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:27.415400 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:27.414155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:49967 (local address 127.0.0.1:36223)
0620 14:35:27.414194 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:27.414196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:27.414389 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:35:27.414495 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:27.414496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:27.414511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:27.414543 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:27.415181 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:27.415182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:27.415182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:27.415187 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:27.415199 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:27.415203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:27.415253 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:27.415306 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:35:27.415321 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:27.416054 21554 catalog_manager.cc:1347] Generated new cluster ID: 76b734c483c54814ab3c678b988db64c
I20260620 14:35:27.416102 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:27.448791 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:27.449013 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:27.455272 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 420302ca839147a2bdfacea357dbf8a5: Generated new TSK 0
I20260620 14:35:27.455495 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:27.457269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21460
2026-06-20T14:35:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:27 dist-test-slave-tc2g krb5kdc[21559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:27 dist-test-slave-tc2g krb5kdc[21559](info): set up 2 sockets
Jun 20 14:35:27 dist-test-slave-tc2g krb5kdc[21559](info): commencing operation
krb5kdc: starting...
W20260620 14:35:28.493561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:28.609071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:28.689320 21572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:28.689584 21572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:28.689626 21572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:28.690761 21572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:28.690831 21572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:28.690858 21572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:28.690888 21572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:28.690912 21572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:28.692358 21572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:28.692658 21572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:28.692878 21572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:28.697656 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:28.697698 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:28.697688 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:28.697924 21572 server_base.cc:1034] running on GCE node
I20260620 14:35:28.698179 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:28.698493 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:28.699671 21572 hybrid_clock.cc:648] HybridClock initialized: now 1781966128699659 us; error 44 us; skew 500 ppm
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:28.709239 21572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:28.709746 21572 webserver.cc:460] Webserver started at http://127.0.0.1:43595/ using document root <none> and password file <none>
I20260620 14:35:28.709988 21572 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:28.710058 21572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:28.710217 21572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:28.711282 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "961ed0974e614578b9e9f10f891294f9"
format_stamp: "Formatted at 2026-06-20 14:35:28 on dist-test-slave-tc2g"
server_key: "b2f89edbe81bda41345dd4b49cebaabd"
server_key_iv: "87b4fecb70c2b4a76161ecdf37c69d80"
server_key_version: "encryptionkey@0"
I20260620 14:35:28.711647 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "961ed0974e614578b9e9f10f891294f9"
format_stamp: "Formatted at 2026-06-20 14:35:28 on dist-test-slave-tc2g"
server_key: "b2f89edbe81bda41345dd4b49cebaabd"
server_key_iv: "87b4fecb70c2b4a76161ecdf37c69d80"
server_key_version: "encryptionkey@0"
I20260620 14:35:28.712957 21572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:28.713729 21587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:28.713896 21572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:28.713987 21572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "961ed0974e614578b9e9f10f891294f9"
format_stamp: "Formatted at 2026-06-20 14:35:28 on dist-test-slave-tc2g"
server_key: "b2f89edbe81bda41345dd4b49cebaabd"
server_key_iv: "87b4fecb70c2b4a76161ecdf37c69d80"
server_key_version: "encryptionkey@0"
I20260620 14:35:28.714073 21572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:28.725929 21572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:28.727751 21572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:28.727906 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:28.731926 21572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33911
I20260620 14:35:28.731957 21639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33911 every 8 connection(s)
I20260620 14:35:28.732306 21572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:28.733177 21640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:28.733645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21572
I20260620 14:35:28.733757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:28.733935 392 external_mini_cluster.cc:1428] Setting key 98d2b4f1c231f06b1e77fe9eb6c18097
I20260620 14:35:28.736240 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9: Bootstrap starting.
I20260620 14:35:28.736763 21640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:28.736977 21640 log.cc:824] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:28.737738 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9: No bootstrap required, opened a new log
I20260620 14:35:28.740260 21640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } }
I20260620 14:35:28.740391 21640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:28.740413 21640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961ed0974e614578b9e9f10f891294f9, State: Initialized, Role: FOLLOWER
I20260620 14:35:28.740540 21640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } }
I20260620 14:35:28.740617 21640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:28.740639 21640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:28.740705 21640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:35:28.741411 21640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } }
I20260620 14:35:28.741559 21640 leader_election.cc:302] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 961ed0974e614578b9e9f10f891294f9; no voters:
I20260620 14:35:28.741730 21640 leader_election.cc:288] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:28.741912 21645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:28.742060 21640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:28.742110 21645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [term 1 LEADER]: Becoming Leader. State: Replica: 961ed0974e614578b9e9f10f891294f9, State: Running, Role: LEADER
I20260620 14:35:28.742246 21645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } }
I20260620 14:35:28.742781 21646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "961ed0974e614578b9e9f10f891294f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } } }
I20260620 14:35:28.742795 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961ed0974e614578b9e9f10f891294f9. Latest consensus state: current_term: 1 leader_uuid: "961ed0974e614578b9e9f10f891294f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961ed0974e614578b9e9f10f891294f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33911 } } }
I20260620 14:35:28.742939 21646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:28.743024 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:28.743250 21649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:28.743706 21649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:28.745493 21649 catalog_manager.cc:1347] Generated new cluster ID: 3a3a762b4e974158a5105c7fd1b7e99d
I20260620 14:35:28.745523 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:28.734418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:46725 (local address 127.0.0.1:33911)
0620 14:35:28.734596 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:28.734604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:28.734627 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:35:28.735024 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:28.735029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:28.735074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:28.735187 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:28.735959 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:28.736949 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:28.738001 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:28.738398 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:28.738479 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:28.741290 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:28.741293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:28.741293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:28.741304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:28.743519 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:28.744014 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:28.744017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:28.744017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:28.744129 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:28.744389 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:28.744390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:28.744390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:28.744675 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:28.744854 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:28.745165 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:35:28.745272 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 14:35:28.745548 21649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:28.746337 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:28.755403 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:28.747677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:42777 (local address 127.0.0.1:33911)
0620 14:35:28.747707 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:28.747709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:28.747815 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:35:28.747869 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:28.747870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:28.747878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:28.747930 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:28.748530 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:28.749527 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:28.750498 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:28.750807 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:28.750861 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:28.752196 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:28.752197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:28.752198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:28.752204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:28.754225 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:28.754623 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:28.754624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:28.754624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:28.754673 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:28.754897 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:28.754898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:28.754898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:28.755008 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:28.755111 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:28.755264 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:35:28.755320 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:35:28.779808 21649 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:28.780594 21649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:28.783949 21649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 961ed0974e614578b9e9f10f891294f9: Generated new TSK 0
I20260620 14:35:28.784132 21649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:28.791743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21572
W20260620 14:35:28.796139 21557 connection.cc:535] client connection to 127.0.0.1:33911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33911): Connection reset by peer (error 104)
2026-06-20T14:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21671](info): set up 2 sockets
Jun 20 14:35:28 dist-test-slave-tc2g krb5kdc[21671](info): commencing operation
krb5kdc: starting...
W20260620 14:35:29.825408 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:29 dist-test-slave-tc2g krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:29.938253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:30.016361 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:30.016607 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:30.016649 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:30.017831 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:30.017903 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:30.017927 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:30.017951 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:30.017978 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:30.019420 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:30.019696 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:30.019918 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:30.024648 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:30.024679 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:30.024648 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:30.024932 21684 server_base.cc:1034] running on GCE node
I20260620 14:35:30.025166 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:30.025473 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:30.026669 21684 hybrid_clock.cc:648] HybridClock initialized: now 1781966130026642 us; error 39 us; skew 500 ppm
Jun 20 14:35:30 dist-test-slave-tc2g krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:30.029017 21684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:30.029521 21684 webserver.cc:460] Webserver started at http://127.0.0.1:41339/ using document root <none> and password file <none>
I20260620 14:35:30.029762 21684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:30.029830 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:30.029963 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:30.030977 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d"
format_stamp: "Formatted at 2026-06-20 14:35:30 on dist-test-slave-tc2g"
server_key: "87508c1051a0cc44c934c9ab0c7eea3a"
server_key_iv: "17dd9f3da4f50457a6b486cea083d93a"
server_key_version: "encryptionkey@0"
I20260620 14:35:30.031335 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d"
format_stamp: "Formatted at 2026-06-20 14:35:30 on dist-test-slave-tc2g"
server_key: "87508c1051a0cc44c934c9ab0c7eea3a"
server_key_iv: "17dd9f3da4f50457a6b486cea083d93a"
server_key_version: "encryptionkey@0"
I20260620 14:35:30.032641 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:30.033401 21699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:30.033627 21684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:30.033725 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d"
format_stamp: "Formatted at 2026-06-20 14:35:30 on dist-test-slave-tc2g"
server_key: "87508c1051a0cc44c934c9ab0c7eea3a"
server_key_iv: "17dd9f3da4f50457a6b486cea083d93a"
server_key_version: "encryptionkey@0"
I20260620 14:35:30.033782 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:30.047043 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:30.048715 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:30.048820 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:30.053557 21684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36125
I20260620 14:35:30.053606 21751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36125 every 8 connection(s)
I20260620 14:35:30.053933 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:30.054889 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:30.057041 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d: Bootstrap starting.
I20260620 14:35:30.057652 21752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:30.057926 21752 log.cc:824] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:30.058586 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d: No bootstrap required, opened a new log
I20260620 14:35:30.060120 21752 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20260620 14:35:30.060273 21752 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:30.060318 21752 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1accf28dd2a4bdd9b4b0ec566959f2d, State: Initialized, Role: FOLLOWER
I20260620 14:35:30.060458 21752 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20260620 14:35:30.060541 21752 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:30.060588 21752 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:30.060637 21752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:30.061151 21752 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20260620 14:35:30.061260 21752 leader_election.cc:302] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1accf28dd2a4bdd9b4b0ec566959f2d; no voters:
I20260620 14:35:30.061470 21752 leader_election.cc:288] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:30.061496 21755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:30.061648 21755 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [term 1 LEADER]: Becoming Leader. State: Replica: c1accf28dd2a4bdd9b4b0ec566959f2d, State: Running, Role: LEADER
I20260620 14:35:30.061707 21752 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:30.061779 21755 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } }
I20260620 14:35:30.062125 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1accf28dd2a4bdd9b4b0ec566959f2d. Latest consensus state: current_term: 1 leader_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } } }
I20260620 14:35:30.062216 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:30.062251 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1accf28dd2a4bdd9b4b0ec566959f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36125 } } }
I20260620 14:35:30.062309 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:30.062469 21763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:30.062938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21684
I20260620 14:35:30.063050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:30.063084 21763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:30.063256 392 external_mini_cluster.cc:1428] Setting key ad7aa63a7b8ae66ee31ee3812654c010
I20260620 14:35:30.064599 21763 catalog_manager.cc:1347] Generated new cluster ID: a93086b34c48465894da43a5a82db28a
I20260620 14:35:30.064652 21763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:30.068246 21763 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:30.068953 21763 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:30.073184 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:30.063686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:56905 (local address 127.0.0.1:36125)
0620 14:35:30.064369 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:30.064377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:30.064405 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:35:30.064514 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:30.064521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:30.067855 (+ 3334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:30.068007 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:30.068206 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:30.069956 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:30.071080 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:30.071642 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:30.071739 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:30.072501 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:30.072504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:30.072505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:30.072720 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:30.072757 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:30.072771 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:30.072852 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:30.072946 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:35:30.072971 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":97,"threads_started":1}
I20260620 14:35:30.074131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:30.074296 21763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1accf28dd2a4bdd9b4b0ec566959f2d: Generated new TSK 0
I20260620 14:35:30.074431 21763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:30.080271 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:30.075275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:40419 (local address 127.0.0.1:36125)
0620 14:35:30.075307 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:30.075309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:30.077617 (+ 2308us) server_negotiation.cc:407] Connection header received
0620 14:35:30.077692 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:30.077693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:30.077714 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:30.077768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:30.077913 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:30.078382 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:30.079085 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:30.079320 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:30.079397 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:30.080047 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:30.080048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:30.080048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:30.080052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:30.080065 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:30.080069 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:30.080120 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:30.080177 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:35:30.080198 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:35:30.082121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21684
W20260620 14:35:30.086259 21669 connection.cc:535] client connection to 127.0.0.1:36125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36125): Connection reset by peer (error 104)
2026-06-20T14:35:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:30 dist-test-slave-tc2g krb5kdc[21783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:30 dist-test-slave-tc2g krb5kdc[21783](info): set up 2 sockets
Jun 20 14:35:30 dist-test-slave-tc2g krb5kdc[21783](info): commencing operation
krb5kdc: starting...
W20260620 14:35:31.117673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:31 dist-test-slave-tc2g krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:35:31.229153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:31.307723 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:31.307909 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:31.307930 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:31.309109 21796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:31.309154 21796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:31.309168 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:31.309181 21796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:31.309193 21796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:31.310694 21796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:31.310895 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:31.311094 21796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:31.315861 21796 server_base.cc:1034] running on GCE node
W20260620 14:35:31.315834 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:31.315927 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:31.315891 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:31.316296 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:31.316610 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:31.317770 21796 hybrid_clock.cc:648] HybridClock initialized: now 1781966131317751 us; error 36 us; skew 500 ppm
Jun 20 14:35:31 dist-test-slave-tc2g krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:31.320102 21796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:35:31.320472 21796 webserver.cc:460] Webserver started at http://127.0.0.1:43411/ using document root <none> and password file <none>
I20260620 14:35:31.320641 21796 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:31.320681 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:31.320782 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:31.321808 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "4f36b81d5ad04ecab53c29208a092ff4"
format_stamp: "Formatted at 2026-06-20 14:35:31 on dist-test-slave-tc2g"
server_key: "f9578c29fa51ce80a880c305b15d95f1"
server_key_iv: "326fd1ddfb0aa6aaabb405f3430c42b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:31.322098 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f36b81d5ad04ecab53c29208a092ff4"
format_stamp: "Formatted at 2026-06-20 14:35:31 on dist-test-slave-tc2g"
server_key: "f9578c29fa51ce80a880c305b15d95f1"
server_key_iv: "326fd1ddfb0aa6aaabb405f3430c42b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:31.323293 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:31.323936 21811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:31.324083 21796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:31.324178 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "4f36b81d5ad04ecab53c29208a092ff4"
format_stamp: "Formatted at 2026-06-20 14:35:31 on dist-test-slave-tc2g"
server_key: "f9578c29fa51ce80a880c305b15d95f1"
server_key_iv: "326fd1ddfb0aa6aaabb405f3430c42b2"
server_key_version: "encryptionkey@0"
I20260620 14:35:31.324230 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:31.327287 21796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:31.328959 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:31.329058 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:31.332624 21796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38633
I20260620 14:35:31.332691 21863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38633 every 8 connection(s)
I20260620 14:35:31.332995 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:31.333576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21796
I20260620 14:35:31.333683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:31.333871 392 external_mini_cluster.cc:1428] Setting key d37da603d07be4aa82aae92f9b77bfdb
I20260620 14:35:31.334159 21864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:31.336494 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:31.334376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:42721 (local address 127.0.0.1:38633)
0620 14:35:31.334569 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:31.334579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:31.334611 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:35:31.335233 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:31.335241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:31.335274 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:31.335421 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:31.336006 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:31.336010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:31.336010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:31.336015 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:31.336058 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:31.336072 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:31.336150 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:31.336185 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:35:31.336323 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260620 14:35:31.337357 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:31.337546 21864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4: Bootstrap starting.
I20260620 14:35:31.338099 21864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:31.338428 21864 log.cc:824] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:31.339178 21864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4: No bootstrap required, opened a new log
I20260620 14:35:31.339727 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:31.338622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:39839 (local address 127.0.0.1:38633)
0620 14:35:31.338714 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:31.338716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:31.338775 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:35:31.338880 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:31.338881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:31.338886 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:31.338945 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:31.339492 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:31.339493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:31.339493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:31.339497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:31.339507 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:31.339510 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:31.339564 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:31.339636 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:35:31.339645 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:35:31.341132 21864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260620 14:35:31.341284 21864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:31.341331 21864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f36b81d5ad04ecab53c29208a092ff4, State: Initialized, Role: FOLLOWER
I20260620 14:35:31.341506 21864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260620 14:35:31.341604 21864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:31.341648 21864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:31.341706 21864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:31.342208 21864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260620 14:35:31.342322 21864 leader_election.cc:302] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f36b81d5ad04ecab53c29208a092ff4; no voters:
I20260620 14:35:31.342495 21864 leader_election.cc:288] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:31.342540 21874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:31.342711 21874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [term 1 LEADER]: Becoming Leader. State: Replica: 4f36b81d5ad04ecab53c29208a092ff4, State: Running, Role: LEADER
I20260620 14:35:31.342775 21864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:31.342832 21874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260620 14:35:31.343212 21875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f36b81d5ad04ecab53c29208a092ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260620 14:35:31.343209 21876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f36b81d5ad04ecab53c29208a092ff4. Latest consensus state: current_term: 1 leader_uuid: "4f36b81d5ad04ecab53c29208a092ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f36b81d5ad04ecab53c29208a092ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260620 14:35:31.343329 21875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:31.343334 21876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:31.343776 21882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:31.344269 21882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:31.345696 21882 catalog_manager.cc:1347] Generated new cluster ID: a8082a57762e44c3b4500198e896d209
I20260620 14:35:31.345746 21882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:31.362823 21882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:31.362990 21882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:31.365705 21882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f36b81d5ad04ecab53c29208a092ff4: Generated new TSK 0
I20260620 14:35:31.365869 21882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:31.376461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21796
2026-06-20T14:35:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:31 dist-test-slave-tc2g krb5kdc[21895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:31 dist-test-slave-tc2g krb5kdc[21895](info): set up 2 sockets
Jun 20 14:35:31 dist-test-slave-tc2g krb5kdc[21895](info): commencing operation
krb5kdc: starting...
W20260620 14:35:32.421397 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:32.536052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:32.615480 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:32.615772 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:32.615814 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:32.617020 21908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:32.617103 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:32.617128 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:32.617154 21908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:32.617187 21908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:32.618674 21908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:32.618942 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:32.619164 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:32.624205 21908 server_base.cc:1034] running on GCE node
W20260620 14:35:32.624157 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:32.624176 21913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:32.624178 21914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:32.624675 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:32.625082 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:32.626281 21908 hybrid_clock.cc:648] HybridClock initialized: now 1781966132626266 us; error 26 us; skew 500 ppm
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:32.628733 21908 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:32.629196 21908 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260620 14:35:32.629372 21908 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:32.629412 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:32.629585 21908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:32.630556 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "81c12941e613471f8cbac3283d4c1995"
format_stamp: "Formatted at 2026-06-20 14:35:32 on dist-test-slave-tc2g"
server_key: "0dc84b22bffcc2791511e957dbb6bbfc"
server_key_iv: "e16b3c90d961b18fb9a1b5f6a4cd4fe5"
server_key_version: "encryptionkey@0"
I20260620 14:35:32.630853 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "81c12941e613471f8cbac3283d4c1995"
format_stamp: "Formatted at 2026-06-20 14:35:32 on dist-test-slave-tc2g"
server_key: "0dc84b22bffcc2791511e957dbb6bbfc"
server_key_iv: "e16b3c90d961b18fb9a1b5f6a4cd4fe5"
server_key_version: "encryptionkey@0"
I20260620 14:35:32.632174 21908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:35:32.632885 21923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:32.633136 21908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:32.633231 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "81c12941e613471f8cbac3283d4c1995"
format_stamp: "Formatted at 2026-06-20 14:35:32 on dist-test-slave-tc2g"
server_key: "0dc84b22bffcc2791511e957dbb6bbfc"
server_key_iv: "e16b3c90d961b18fb9a1b5f6a4cd4fe5"
server_key_version: "encryptionkey@0"
I20260620 14:35:32.633286 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:32.657636 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:32.660070 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:32.660187 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:32.664093 21908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45159
I20260620 14:35:32.664147 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45159 every 8 connection(s)
I20260620 14:35:32.664479 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:32.665359 21976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:32.667613 21976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995: Bootstrap starting.
I20260620 14:35:32.668195 21976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:32.668473 21976 log.cc:824] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:32.669150 21976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995: No bootstrap required, opened a new log
I20260620 14:35:32.670778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 21908
I20260620 14:35:32.670862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:32.670809 21976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } }
I20260620 14:35:32.671018 21976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:32.671065 392 external_mini_cluster.cc:1428] Setting key 27e2610895d6e8533f3bc37df19c91d6
I20260620 14:35:32.671072 21976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c12941e613471f8cbac3283d4c1995, State: Initialized, Role: FOLLOWER
I20260620 14:35:32.671247 21976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } }
I20260620 14:35:32.671352 21976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:32.671412 21976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:32.671468 21976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:32.672513 21976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } }
I20260620 14:35:32.672633 21976 leader_election.cc:302] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c12941e613471f8cbac3283d4c1995; no voters:
I20260620 14:35:32.672798 21976 leader_election.cc:288] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:32.672860 21981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:32.673038 21981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [term 1 LEADER]: Becoming Leader. State: Replica: 81c12941e613471f8cbac3283d4c1995, State: Running, Role: LEADER
I20260620 14:35:32.673049 21976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:32.673198 21981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } }
I20260620 14:35:32.673753 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c12941e613471f8cbac3283d4c1995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } } }
I20260620 14:35:32.673898 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:32.674505 21990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:32.675149 21990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:32.674500 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c12941e613471f8cbac3283d4c1995. Latest consensus state: current_term: 1 leader_uuid: "81c12941e613471f8cbac3283d4c1995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c12941e613471f8cbac3283d4c1995" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45159 } } }
I20260620 14:35:32.675859 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[21895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:32.678961 21990 catalog_manager.cc:1347] Generated new cluster ID: ea02926568c74939a329d4f678d8628b
I20260620 14:35:32.679021 21990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:32.682327 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:32.671508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47727 (local address 127.0.98.62:45159)
0620 14:35:32.671735 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:32.671745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:32.671782 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:35:32.671935 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:32.671945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:32.672004 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:32.672192 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:32.672789 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:32.674343 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:32.675480 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:32.675963 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:32.676069 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:32.678917 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:32.678921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:32.678921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:32.678931 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:32.680646 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:32.681041 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:32.681045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:32.681045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:32.681158 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:32.681394 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:32.681395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:32.681395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:32.681678 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:32.681853 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:32.681978 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:35:32.682181 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
I20260620 14:35:32.683255 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:32.692651 21990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:32.693257 21990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:32.696096 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:32.684572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43723 (local address 127.0.98.62:45159)
0620 14:35:32.684615 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:32.684617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:32.689588 (+ 4971us) server_negotiation.cc:407] Connection header received
0620 14:35:32.689669 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:32.689670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:32.689680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:32.689746 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:32.690264 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:32.690753 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:32.691458 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:32.691676 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:32.691761 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:32.693193 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:32.693194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:32.693194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:32.693202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:32.694894 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:32.695324 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:32.695325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:32.695326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:32.695376 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:32.695609 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:32.695610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:32.695611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:32.695720 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:32.695822 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:32.695953 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:35:32.696009 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:32.697242 21990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c12941e613471f8cbac3283d4c1995: Generated new TSK 0
I20260620 14:35:32.697404 21990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:32.703994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 21908
W20260620 14:35:32.708212 21893 connection.cc:535] client connection to 127.0.98.62:45159 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45159): Connection reset by peer (error 104)
2026-06-20T14:35:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[22007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[22007](info): set up 2 sockets
Jun 20 14:35:32 dist-test-slave-tc2g krb5kdc[22007](info): commencing operation
krb5kdc: starting...
W20260620 14:35:33.733520 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:33 dist-test-slave-tc2g krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:33.844434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:33.923318 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:33.923509 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:33.923530 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:33.924687 22020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:33.924728 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:33.924743 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:33.924783 22020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:33.924814 22020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:33.926259 22020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:33.926497 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:33.926725 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:33.931468 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:33.931509 22020 server_base.cc:1034] running on GCE node
W20260620 14:35:33.931595 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:33.931517 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:33.931924 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:33.932180 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:33.933327 22020 hybrid_clock.cc:648] HybridClock initialized: now 1781966133933303 us; error 49 us; skew 500 ppm
Jun 20 14:35:33 dist-test-slave-tc2g krb5kdc[22007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:33.939119 22020 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:33.939515 22020 webserver.cc:460] Webserver started at http://127.0.98.62:37857/ using document root <none> and password file <none>
I20260620 14:35:33.939708 22020 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:33.939749 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:33.939837 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:33.940851 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "28c29dbbe1cf46d1b07cb34e2747c62e"
format_stamp: "Formatted at 2026-06-20 14:35:33 on dist-test-slave-tc2g"
server_key: "f13baec430f59810b2e8676ecdfbba9a"
server_key_iv: "21d93c9e297d12c9148b5feeb3b4bfd7"
server_key_version: "encryptionkey@0"
I20260620 14:35:33.941147 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "28c29dbbe1cf46d1b07cb34e2747c62e"
format_stamp: "Formatted at 2026-06-20 14:35:33 on dist-test-slave-tc2g"
server_key: "f13baec430f59810b2e8676ecdfbba9a"
server_key_iv: "21d93c9e297d12c9148b5feeb3b4bfd7"
server_key_version: "encryptionkey@0"
I20260620 14:35:33.942534 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:33.943244 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:33.943441 22020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:33.943553 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "28c29dbbe1cf46d1b07cb34e2747c62e"
format_stamp: "Formatted at 2026-06-20 14:35:33 on dist-test-slave-tc2g"
server_key: "f13baec430f59810b2e8676ecdfbba9a"
server_key_iv: "21d93c9e297d12c9148b5feeb3b4bfd7"
server_key_version: "encryptionkey@0"
I20260620 14:35:33.943637 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:33.951207 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:33.953243 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:33.953411 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:33.957175 22020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260620 14:35:33.957237 22087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260620 14:35:33.957656 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:33.958508 22088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:33.958738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22020
I20260620 14:35:33.958865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:33.959067 392 external_mini_cluster.cc:1428] Setting key db1184ee1adfb23a98c24d44e7d190b0
I20260620 14:35:33.962080 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e: Bootstrap starting.
I20260620 14:35:33.962792 22088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:33.963083 22088 log.cc:824] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:33.963783 22088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e: No bootstrap required, opened a new log
I20260620 14:35:33.965281 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:33.959589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48261 (local address 127.0.98.62:43377)
0620 14:35:33.959810 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:33.959820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:33.959854 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:35:33.960329 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:33.960338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:33.960400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:33.960550 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:33.960743 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:33.962159 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:33.963348 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:33.963849 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:33.963988 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:33.964709 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:33.964713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:33.964713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:33.964872 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:33.964898 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:33.964909 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:33.964996 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:33.965094 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:35:33.965122 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20260620 14:35:33.965992 22088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 14:35:33.966164 22088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:33.966217 22088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c29dbbe1cf46d1b07cb34e2747c62e, State: Initialized, Role: FOLLOWER
I20260620 14:35:33.966298 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:33.966338 22088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 14:35:33.966410 22088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:33.966444 22088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:33.966485 22088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:33.967149 22088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 14:35:33.967257 22088 leader_election.cc:302] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c29dbbe1cf46d1b07cb34e2747c62e; no voters:
I20260620 14:35:33.967424 22088 leader_election.cc:288] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:33.967589 22098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:33.967676 22088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:33.967803 22098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [term 1 LEADER]: Becoming Leader. State: Replica: 28c29dbbe1cf46d1b07cb34e2747c62e, State: Running, Role: LEADER
I20260620 14:35:33.968003 22098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 14:35:33.968506 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260620 14:35:33.968516 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c29dbbe1cf46d1b07cb34e2747c62e. Latest consensus state: current_term: 1 leader_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c29dbbe1cf46d1b07cb34e2747c62e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260620 14:35:33.968611 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:33.968616 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:33.968822 22107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:33.969360 22107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:33.971074 22107 catalog_manager.cc:1347] Generated new cluster ID: 0d4ec3782bc4472b8047ce17bb229506
I20260620 14:35:33.971159 22107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:33.971469 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:33.967928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54935 (local address 127.0.98.62:43377)
0620 14:35:33.967962 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:33.967964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:33.968078 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:35:33.968138 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:33.968138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:33.968148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:33.968202 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:33.968402 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:33.969111 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:33.970158 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:33.970376 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:33.970462 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:33.971232 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:33.971233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:33.971233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:33.971238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:33.971250 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:33.971254 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:33.971304 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:33.971370 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:35:33.971380 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:33.984114 22107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:33.984623 22107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:33.992081 22107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c29dbbe1cf46d1b07cb34e2747c62e: Generated new TSK 0
I20260620 14:35:33.992274 22107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:33.995435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22020
W20260620 14:35:33.999804 22005 connection.cc:535] client connection to 127.0.98.62:43377 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43377): Connection reset by peer (error 104)
2026-06-20T14:35:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:34 dist-test-slave-tc2g krb5kdc[22119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:34 dist-test-slave-tc2g krb5kdc[22119](info): set up 2 sockets
Jun 20 14:35:34 dist-test-slave-tc2g krb5kdc[22119](info): commencing operation
krb5kdc: starting...
W20260620 14:35:35.025964 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:35 dist-test-slave-tc2g krb5kdc[22119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:35.143765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:35.222349 22132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:35.222591 22132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:35.222626 22132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:35.223735 22132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:35.223803 22132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:35.223827 22132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:35.223857 22132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:35.223881 22132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:35.225298 22132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:35.225577 22132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:35.225802 22132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:35.230384 22140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:35.230389 22137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:35.230393 22138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:35.230680 22132 server_base.cc:1034] running on GCE node
I20260620 14:35:35.230935 22132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:35.231215 22132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:35.232391 22132 hybrid_clock.cc:648] HybridClock initialized: now 1781966135232373 us; error 37 us; skew 500 ppm
Jun 20 14:35:35 dist-test-slave-tc2g krb5kdc[22119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:35.234956 22132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:35.235401 22132 webserver.cc:460] Webserver started at http://127.0.98.62:39445/ using document root <none> and password file <none>
I20260620 14:35:35.235625 22132 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:35.235692 22132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:35.235841 22132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:35.236845 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "0ce3873a37134cab8704f193ae22f390"
format_stamp: "Formatted at 2026-06-20 14:35:35 on dist-test-slave-tc2g"
server_key: "7a7ecd9980519c796e5f3c7c5062c473"
server_key_iv: "449b712193c8a8b4430c44f323cefe52"
server_key_version: "encryptionkey@0"
I20260620 14:35:35.237228 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ce3873a37134cab8704f193ae22f390"
format_stamp: "Formatted at 2026-06-20 14:35:35 on dist-test-slave-tc2g"
server_key: "7a7ecd9980519c796e5f3c7c5062c473"
server_key_iv: "449b712193c8a8b4430c44f323cefe52"
server_key_version: "encryptionkey@0"
I20260620 14:35:35.238569 22132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:35:35.239324 22147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:35.239506 22132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:35.239616 22132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "0ce3873a37134cab8704f193ae22f390"
format_stamp: "Formatted at 2026-06-20 14:35:35 on dist-test-slave-tc2g"
server_key: "7a7ecd9980519c796e5f3c7c5062c473"
server_key_iv: "449b712193c8a8b4430c44f323cefe52"
server_key_version: "encryptionkey@0"
I20260620 14:35:35.239693 22132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:35.242951 22132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:35.244822 22132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:35.245003 22132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:35.249080 22132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40769
I20260620 14:35:35.249109 22199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40769 every 8 connection(s)
I20260620 14:35:35.249475 22132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:35.250391 22200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:35.252501 22200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: Bootstrap starting.
I20260620 14:35:35.253062 22200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:35.253342 22200 log.cc:824] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:35.253993 22200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: No bootstrap required, opened a new log
I20260620 14:35:35.255508 22200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } }
I20260620 14:35:35.255637 22200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:35.255659 22200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce3873a37134cab8704f193ae22f390, State: Initialized, Role: FOLLOWER
I20260620 14:35:35.255827 22200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } }
I20260620 14:35:35.255887 22200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:35.255926 22200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:35.255975 22200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:35.256485 22200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } }
I20260620 14:35:35.256590 22200 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce3873a37134cab8704f193ae22f390; no voters:
I20260620 14:35:35.256765 22200 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:35.256795 22203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:35.256980 22203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce3873a37134cab8704f193ae22f390, State: Running, Role: LEADER
I20260620 14:35:35.257014 22200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:35.257112 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } }
I20260620 14:35:35.257503 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce3873a37134cab8704f193ae22f390. Latest consensus state: current_term: 1 leader_uuid: "0ce3873a37134cab8704f193ae22f390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } } }
I20260620 14:35:35.257581 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:35.257884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22132
I20260620 14:35:35.257982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:35.258143 392 external_mini_cluster.cc:1428] Setting key 5054e7b3aa7bb653447516567a48ee59
I20260620 14:35:35.257642 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce3873a37134cab8704f193ae22f390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce3873a37134cab8704f193ae22f390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40769 } } }
I20260620 14:35:35.258314 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390 [sys.catalog]: This master's current role is: LEADER
W20260620 14:35:35.259195 22220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:35.259263 22220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:35.259321 22211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:35.259456 22211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:35.260629 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:35.258773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37127 (local address 127.0.98.62:40769)
0620 14:35:35.258943 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:35.258952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:35.258987 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:35:35.259112 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:35.259119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:35.259149 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:35.259303 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:35.260079 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:35.260082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:35.260082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:35.260086 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:35.260137 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:35.260148 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:35.260224 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:35.260435 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:35:35.260476 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260620 14:35:35.260807 22211 catalog_manager.cc:1347] Generated new cluster ID: 3ab7233054fe4e28a0bd810d452d3bbb
I20260620 14:35:35.260851 22211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:35.261785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:35.264318 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:35.263247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58449 (local address 127.0.98.62:40769)
0620 14:35:35.263274 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:35.263277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:35.263421 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:35:35.263483 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:35.263483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:35.263501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:35.263555 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:35.264095 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:35.264103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:35.264103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:35.264106 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:35.264120 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:35.264123 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:35.264181 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:35.264254 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:35:35.264265 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 14:35:35.270593 22211 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:35.270733 22211 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:35.276619 22211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce3873a37134cab8704f193ae22f390: Generated new TSK 0
I20260620 14:35:35.276778 22211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:35.282671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22132
2026-06-20T14:35:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:35 dist-test-slave-tc2g krb5kdc[22231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:35 dist-test-slave-tc2g krb5kdc[22231](info): set up 2 sockets
Jun 20 14:35:35 dist-test-slave-tc2g krb5kdc[22231](info): commencing operation
krb5kdc: starting...
W20260620 14:35:36.305513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:36.415814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:36.494733 22244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:36.494961 22244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:36.494998 22244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:36.496117 22244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:36.496184 22244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:36.496209 22244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:36.496237 22244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:36.496260 22244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:36.497704 22244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:36.497957 22244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:36.498183 22244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:36.502794 22252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:36.502758 22250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:36.502810 22249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:36.503208 22244 server_base.cc:1034] running on GCE node
I20260620 14:35:36.503412 22244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:36.503645 22244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:36.504779 22244 hybrid_clock.cc:648] HybridClock initialized: now 1781966136504764 us; error 31 us; skew 500 ppm
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:36.507043 22244 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:36.507445 22244 webserver.cc:460] Webserver started at http://127.0.98.62:39747/ using document root <none> and password file <none>
I20260620 14:35:36.507671 22244 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:36.507740 22244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:36.507896 22244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:36.508939 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "c3d5d5d2748d4765bf0c0682248cdbb0"
format_stamp: "Formatted at 2026-06-20 14:35:36 on dist-test-slave-tc2g"
server_key: "bc528522123eaaed26714f28afe6d512"
server_key_iv: "a3709ec34cc9a207dc54cc2a94f57bd9"
server_key_version: "encryptionkey@0"
I20260620 14:35:36.509289 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3d5d5d2748d4765bf0c0682248cdbb0"
format_stamp: "Formatted at 2026-06-20 14:35:36 on dist-test-slave-tc2g"
server_key: "bc528522123eaaed26714f28afe6d512"
server_key_iv: "a3709ec34cc9a207dc54cc2a94f57bd9"
server_key_version: "encryptionkey@0"
I20260620 14:35:36.510593 22244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:36.511272 22259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:36.511426 22244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:36.511520 22244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "c3d5d5d2748d4765bf0c0682248cdbb0"
format_stamp: "Formatted at 2026-06-20 14:35:36 on dist-test-slave-tc2g"
server_key: "bc528522123eaaed26714f28afe6d512"
server_key_iv: "a3709ec34cc9a207dc54cc2a94f57bd9"
server_key_version: "encryptionkey@0"
I20260620 14:35:36.511606 22244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:36.526680 22244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:36.528435 22244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:36.528604 22244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:36.532495 22244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46825
I20260620 14:35:36.532531 22311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46825 every 8 connection(s)
I20260620 14:35:36.532876 22244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:36.533881 22312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:36.536087 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0: Bootstrap starting.
I20260620 14:35:36.536659 22312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:36.536937 22312 log.cc:824] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:36.537827 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0: No bootstrap required, opened a new log
I20260620 14:35:36.539435 22312 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260620 14:35:36.539573 22312 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:36.539597 22312 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d5d5d2748d4765bf0c0682248cdbb0, State: Initialized, Role: FOLLOWER
I20260620 14:35:36.539772 22312 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260620 14:35:36.539837 22312 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:36.539875 22312 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:36.539923 22312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:36.540203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22244
I20260620 14:35:36.540318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:36.540504 392 external_mini_cluster.cc:1428] Setting key 9678af08381480c70c5b650285ccff38
I20260620 14:35:36.540583 22312 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260620 14:35:36.540688 22312 leader_election.cc:302] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d5d5d2748d4765bf0c0682248cdbb0; no voters:
I20260620 14:35:36.540858 22312 leader_election.cc:288] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:36.540910 22315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:36.541150 22315 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [term 1 LEADER]: Becoming Leader. State: Replica: c3d5d5d2748d4765bf0c0682248cdbb0, State: Running, Role: LEADER
I20260620 14:35:36.541270 22312 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:36.541281 22315 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } }
I20260620 14:35:36.541659 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } } }
I20260620 14:35:36.541805 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:36.541998 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d5d5d2748d4765bf0c0682248cdbb0. Latest consensus state: current_term: 1 leader_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d5d5d2748d4765bf0c0682248cdbb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46825 } } }
I20260620 14:35:36.542049 22321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:36.542830 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:36.543313 22321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:36.545835 22321 catalog_manager.cc:1347] Generated new cluster ID: 9ed89d2f3db647f6a1281513da857e34
I20260620 14:35:36.545895 22321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:36.552778 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:36.541007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34385 (local address 127.0.98.62:46825)
0620 14:35:36.541507 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:36.541517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:36.541553 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:35:36.541673 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:36.541687 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:36.541745 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:36.541893 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:36.542820 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:36.545235 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:36.546247 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:36.546662 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:36.546795 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:36.549117 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:36.549121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:36.549121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:36.549133 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:36.551269 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:36.551663 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:36.551666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:36.551666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:36.551765 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:36.551972 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:36.551973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:36.551974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:36.552203 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:36.552364 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:36.552461 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:35:36.552614 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":109,"threads_started":1}
I20260620 14:35:36.553668 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:36.560458 22321 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:36.561211 22321 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:36.567346 22318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:36.554973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50655 (local address 127.0.98.62:46825)
0620 14:35:36.555010 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:36.555012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:36.560284 (+ 5272us) server_negotiation.cc:407] Connection header received
0620 14:35:36.560354 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:36.560355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:36.560369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:36.560444 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:36.561096 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:36.561875 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:36.562829 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:36.563158 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:36.563232 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:36.564553 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:36.564554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:36.564555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:36.564561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:36.566230 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:36.566591 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:36.566592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:36.566592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:36.566652 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:36.566838 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:36.566839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:36.566839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:36.566964 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:36.567073 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:36.567166 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:35:36.567255 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:35:36.568867 22321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d5d5d2748d4765bf0c0682248cdbb0: Generated new TSK 0
I20260620 14:35:36.569036 22321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:36.572691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22244
2026-06-20T14:35:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22343](info): set up 2 sockets
Jun 20 14:35:36 dist-test-slave-tc2g krb5kdc[22343](info): commencing operation
krb5kdc: starting...
W20260620 14:35:37.609575 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:37 dist-test-slave-tc2g krb5kdc[22343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:37.720494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:37.798573 22356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:37.798815 22356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:37.798854 22356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:37.799991 22356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:37.800057 22356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:37.800081 22356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:37.800108 22356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:37.800140 22356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:37.801589 22356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:37.801872 22356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:37.802098 22356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:37.806769 22361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:37.806895 22356 server_base.cc:1034] running on GCE node
W20260620 14:35:37.807005 22364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:37.807060 22362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:37.807402 22356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:37.807723 22356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:37.808864 22356 hybrid_clock.cc:648] HybridClock initialized: now 1781966137808855 us; error 33 us; skew 500 ppm
Jun 20 14:35:37 dist-test-slave-tc2g krb5kdc[22343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:37.811282 22356 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:37.811702 22356 webserver.cc:460] Webserver started at http://127.0.98.62:44425/ using document root <none> and password file <none>
I20260620 14:35:37.811920 22356 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:37.811987 22356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:37.812134 22356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:37.813170 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "5c9b337a92e2481f95d45eaa4fb368f1"
format_stamp: "Formatted at 2026-06-20 14:35:37 on dist-test-slave-tc2g"
server_key: "81c180f1b5b92fecc745fa792f1ca0d3"
server_key_iv: "30364f1d3708a064e4661734cc149251"
server_key_version: "encryptionkey@0"
I20260620 14:35:37.813563 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c9b337a92e2481f95d45eaa4fb368f1"
format_stamp: "Formatted at 2026-06-20 14:35:37 on dist-test-slave-tc2g"
server_key: "81c180f1b5b92fecc745fa792f1ca0d3"
server_key_iv: "30364f1d3708a064e4661734cc149251"
server_key_version: "encryptionkey@0"
I20260620 14:35:37.814846 22356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:35:37.815521 22371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:37.815680 22356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:37.815775 22356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "5c9b337a92e2481f95d45eaa4fb368f1"
format_stamp: "Formatted at 2026-06-20 14:35:37 on dist-test-slave-tc2g"
server_key: "81c180f1b5b92fecc745fa792f1ca0d3"
server_key_iv: "30364f1d3708a064e4661734cc149251"
server_key_version: "encryptionkey@0"
I20260620 14:35:37.815858 22356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:37.828994 22356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:37.831284 22356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:37.831444 22356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:37.835496 22356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44315
I20260620 14:35:37.835529 22423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44315 every 8 connection(s)
I20260620 14:35:37.835863 22356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:37.836706 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:37.839074 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1: Bootstrap starting.
I20260620 14:35:37.839639 22424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:37.839915 22424 log.cc:824] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:37.840569 22424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1: No bootstrap required, opened a new log
I20260620 14:35:37.842195 22424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } }
I20260620 14:35:37.842321 22424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:37.842366 22424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9b337a92e2481f95d45eaa4fb368f1, State: Initialized, Role: FOLLOWER
I20260620 14:35:37.842523 22424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } }
I20260620 14:35:37.842585 22424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:37.842607 22424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:37.842633 22424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:37.843149 22424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } }
I20260620 14:35:37.843250 22424 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9b337a92e2481f95d45eaa4fb368f1; no voters:
I20260620 14:35:37.843438 22424 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:37.843502 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:37.843716 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9b337a92e2481f95d45eaa4fb368f1, State: Running, Role: LEADER
I20260620 14:35:37.843761 22424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:37.843869 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } }
I20260620 14:35:37.844506 22428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } } }
I20260620 14:35:37.844513 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9b337a92e2481f95d45eaa4fb368f1. Latest consensus state: current_term: 1 leader_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b337a92e2481f95d45eaa4fb368f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44315 } } }
I20260620 14:35:37.844602 22428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:37.844609 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:37.844805 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:37.844900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22356
I20260620 14:35:37.845014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:37.845229 392 external_mini_cluster.cc:1428] Setting key abebaadb9f9305c6ed6fd05305368af9
I20260620 14:35:37.845304 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:37.846930 22436 catalog_manager.cc:1347] Generated new cluster ID: 9170239142264a6db3d86da0caf265ad
I20260620 14:35:37.846982 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:37.851411 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:37.845799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40939 (local address 127.0.98.62:44315)
0620 14:35:37.846177 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:37.846184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:37.846210 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:35:37.846296 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:37.846301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:37.846341 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:37.846477 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:37.846728 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:37.848073 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:37.849145 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:37.849802 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:37.849904 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:37.850566 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:37.850570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:37.850570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:37.850772 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:37.850816 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:37.850829 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:37.850900 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:37.850971 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:35:37.851204 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":77,"threads_started":1}
I20260620 14:35:37.852320 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:37.861828 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:37.853916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51425 (local address 127.0.98.62:44315)
0620 14:35:37.853976 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:37.853977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:37.857591 (+ 3614us) server_negotiation.cc:407] Connection header received
0620 14:35:37.857618 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:37.857619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:37.857628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:37.857696 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:37.858835 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:37.859601 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:37.860541 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:37.860857 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:37.860925 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:37.861563 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:37.861564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:37.861565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:37.861568 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:37.861589 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:37.861594 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:37.861650 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:37.861704 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:35:37.861727 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:35:37.871577 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:37.872282 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:37.875813 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9b337a92e2481f95d45eaa4fb368f1: Generated new TSK 0
I20260620 14:35:37.875979 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:37.878033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22356
2026-06-20T14:35:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:37 dist-test-slave-tc2g krb5kdc[22455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:37 dist-test-slave-tc2g krb5kdc[22455](info): set up 2 sockets
Jun 20 14:35:37 dist-test-slave-tc2g krb5kdc[22455](info): commencing operation
krb5kdc: starting...
W20260620 14:35:38.905622 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:38 dist-test-slave-tc2g krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:39.017287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:39.098162 22468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:39.098413 22468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:39.098454 22468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:39.099658 22468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:39.099723 22468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:39.099749 22468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:39.099781 22468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:39.099807 22468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:39.101274 22468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:39.101576 22468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:39.101801 22468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:39.106519 22473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:39.106578 22468 server_base.cc:1034] running on GCE node
W20260620 14:35:39.106544 22474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:39.106534 22476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:39.106936 22468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:39.107263 22468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:39.108440 22468 hybrid_clock.cc:648] HybridClock initialized: now 1781966139108430 us; error 39 us; skew 500 ppm
Jun 20 14:35:39 dist-test-slave-tc2g krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:39.114127 22468 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:39.114594 22468 webserver.cc:460] Webserver started at http://127.0.98.62:33051/ using document root <none> and password file <none>
I20260620 14:35:39.114811 22468 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:39.114877 22468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:39.115032 22468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:39.116051 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "41f478b452d04809b5bb7c2e05d1cf3f"
format_stamp: "Formatted at 2026-06-20 14:35:39 on dist-test-slave-tc2g"
server_key: "80fbf1f2fda0c840c6508df93551b9e1"
server_key_iv: "452e67acfc4b9c400f6237eb89c89346"
server_key_version: "encryptionkey@0"
I20260620 14:35:39.116405 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "41f478b452d04809b5bb7c2e05d1cf3f"
format_stamp: "Formatted at 2026-06-20 14:35:39 on dist-test-slave-tc2g"
server_key: "80fbf1f2fda0c840c6508df93551b9e1"
server_key_iv: "452e67acfc4b9c400f6237eb89c89346"
server_key_version: "encryptionkey@0"
I20260620 14:35:39.117728 22468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:35:39.118443 22483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:39.118634 22468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:35:39.118734 22468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "41f478b452d04809b5bb7c2e05d1cf3f"
format_stamp: "Formatted at 2026-06-20 14:35:39 on dist-test-slave-tc2g"
server_key: "80fbf1f2fda0c840c6508df93551b9e1"
server_key_iv: "452e67acfc4b9c400f6237eb89c89346"
server_key_version: "encryptionkey@0"
I20260620 14:35:39.118805 22468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:39.122150 22468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:39.123893 22468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:39.124050 22468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:39.127997 22468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42655
I20260620 14:35:39.128028 22535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42655 every 8 connection(s)
I20260620 14:35:39.128346 22468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:39.129267 22536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:39.131333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22468
I20260620 14:35:39.131448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:39.131425 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f: Bootstrap starting.
I20260620 14:35:39.131659 392 external_mini_cluster.cc:1428] Setting key aad1dbd8d78ae26aec7aa7d31f7b93cb
I20260620 14:35:39.132093 22536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:39.132428 22536 log.cc:824] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:39.133172 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f: No bootstrap required, opened a new log
I20260620 14:35:39.134127 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:39.132133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34635 (local address 127.0.98.62:42655)
0620 14:35:39.132314 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:39.132321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:39.132346 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:35:39.132793 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:39.132799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:39.132830 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:39.132965 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:39.133699 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:39.133702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:39.133703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:39.133706 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:39.133744 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:39.133754 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:39.133820 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:39.133884 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:35:39.133940 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260620 14:35:39.135046 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:39.135778 22536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } }
I20260620 14:35:39.135957 22536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:39.135990 22536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f478b452d04809b5bb7c2e05d1cf3f, State: Initialized, Role: FOLLOWER
I20260620 14:35:39.136111 22536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } }
I20260620 14:35:39.136220 22536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:39.136276 22536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:39.136338 22536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:39.137040 22536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } }
I20260620 14:35:39.137156 22536 leader_election.cc:302] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f478b452d04809b5bb7c2e05d1cf3f; no voters:
I20260620 14:35:39.137307 22536 leader_election.cc:288] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:39.137362 22546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:39.137567 22536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:39.137554 22546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [term 1 LEADER]: Becoming Leader. State: Replica: 41f478b452d04809b5bb7c2e05d1cf3f, State: Running, Role: LEADER
I20260620 14:35:39.137705 22546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } }
I20260620 14:35:39.138036 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:39.136626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32999 (local address 127.0.98.62:42655)
0620 14:35:39.136664 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:39.136666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:39.136792 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:35:39.136855 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:39.136856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:39.136859 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:39.136902 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:39.137514 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:39.137515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:39.137515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:39.137520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:39.137531 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:39.137535 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:39.137564 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:39.137693 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:35:39.137705 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:39.138242 22548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f478b452d04809b5bb7c2e05d1cf3f. Latest consensus state: current_term: 1 leader_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } }
I20260620 14:35:39.138345 22548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:39.138659 22547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f478b452d04809b5bb7c2e05d1cf3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } }
I20260620 14:35:39.138733 22547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:39.138844 22555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:39.139307 22555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:39.140681 22555 catalog_manager.cc:1347] Generated new cluster ID: ebae9566c27b411db2b06d952f5f49a0
I20260620 14:35:39.140745 22555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:39.154629 22555 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:39.154790 22555 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:39.162586 22555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f478b452d04809b5bb7c2e05d1cf3f: Generated new TSK 0
I20260620 14:35:39.162760 22555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:39.169195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22468
2026-06-20T14:35:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (23496 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:39 dist-test-slave-tc2g krb5kdc[22567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:39 dist-test-slave-tc2g krb5kdc[22567](info): set up 2 sockets
Jun 20 14:35:39 dist-test-slave-tc2g krb5kdc[22567](info): commencing operation
krb5kdc: starting...
W20260620 14:35:40.205533 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:40.319664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:40.398259 22580 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:40.398468 22580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:40.398486 22580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:40.398506 22580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:40.399663 22580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:40.399703 22580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:40.399717 22580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:40.399729 22580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:40.399780 22580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:40.401222 22580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:40.401500 22580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:40.401733 22580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:40.406451 22585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:40.406474 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:40.406474 22586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:40.406646 22580 server_base.cc:1034] running on GCE node
I20260620 14:35:40.406852 22580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:40.407081 22580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:40.408253 22580 hybrid_clock.cc:648] HybridClock initialized: now 1781966140408243 us; error 32 us; skew 500 ppm
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:40.410686 22580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:40.411053 22580 webserver.cc:460] Webserver started at http://127.0.98.62:39335/ using document root <none> and password file <none>
I20260620 14:35:40.411245 22580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:40.411288 22580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:40.411373 22580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:40.412350 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "41aeef117bad4afdab5f6539dc2bd057"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "188fc9c3ab1f9ae0ab96a904cb43d343"
server_key_iv: "2fb7645699264b71c2995642c3cafb44"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.412650 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "41aeef117bad4afdab5f6539dc2bd057"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "188fc9c3ab1f9ae0ab96a904cb43d343"
server_key_iv: "2fb7645699264b71c2995642c3cafb44"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.413976 22580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:35:40.414646 22595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:40.414829 22580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:40.414918 22580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "41aeef117bad4afdab5f6539dc2bd057"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "188fc9c3ab1f9ae0ab96a904cb43d343"
server_key_iv: "2fb7645699264b71c2995642c3cafb44"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.414992 22580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:40.426659 22580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:40.428448 22580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:40.428604 22580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:40.432520 22580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40239
I20260620 14:35:40.432556 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40239 every 8 connection(s)
I20260620 14:35:40.432956 22580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:40.434011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22580
I20260620 14:35:40.434048 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:40.434120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:40.434326 392 external_mini_cluster.cc:1428] Setting key 32a5e3e98135b0ca81bc832ee169f969
I20260620 14:35:40.436298 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:40.448361 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.436978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.0.1:55574)
0620 14:35:40.437200 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.437236 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.437833 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.438110 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.438112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.438386 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.438535 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.438539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.439532 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.439533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.440221 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.440223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.440765 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.440766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.440827 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.441347 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.441355 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.444686 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.446894 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.446900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.446901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.447153 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.447426 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.447427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.447427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.447501 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.447952 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.447957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.448009 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.448090 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.448209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260620 14:35:40.448480 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.437008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.0.98.62:40239)
0620 14:35:40.437184 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.437194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.437333 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:35:40.437995 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.437997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.438013 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.438103 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.438640 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.439428 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.440321 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.440620 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.440758 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.444771 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.444777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.444778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.444784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.446803 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.447240 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.447243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.447244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.447351 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.447565 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.447566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.447566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.447872 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.447966 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.448137 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.448297 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260620 14:35:40.449162 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:40.449550 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.449403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46108)
0620 14:35:40.449486 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.449533 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 14:35:40.449707 22596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
I20260620 14:35:40.450130 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.450044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46120)
0620 14:35:40.450092 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.450115 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:35:40.450248 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:35:40.458406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40173
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:40.499454 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } attempt: 1
I20260620 14:35:40.499938 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.499806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46122)
0620 14:35:40.499867 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.499909 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:35:40.500308 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.500217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46124)
0620 14:35:40.500260 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.500284 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:35:40.500411 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
W20260620 14:35:40.538720 22655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:40.538949 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:40.538980 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:40.539022 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:40.540109 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:40.540177 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:40.540202 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:40.540222 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:40.540251 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:40.542806 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:40.543073 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:40.543329 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:40.549914 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:40.549927 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:40.549957 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:40.551090 22655 server_base.cc:1034] running on GCE node
I20260620 14:35:40.551394 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:40.551779 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:40.553110 22655 hybrid_clock.cc:648] HybridClock initialized: now 1781966140553092 us; error 28 us; skew 500 ppm
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:40.556037 22655 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:35:40.556608 22655 webserver.cc:460] Webserver started at http://127.0.98.61:35297/ using document root <none> and password file <none>
I20260620 14:35:40.556854 22655 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:40.556923 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:40.557062 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:40.558190 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data/instance:
uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "8e53db8f77fa041ea451dc820df5777a"
server_key_iv: "7a249c378df83703528d943b501e4548"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.558557 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal/instance:
uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "8e53db8f77fa041ea451dc820df5777a"
server_key_iv: "7a249c378df83703528d943b501e4548"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.560104 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:40.561002 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:40.561512 22655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:40.561599 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "8e53db8f77fa041ea451dc820df5777a"
server_key_iv: "7a249c378df83703528d943b501e4548"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.561661 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:40.568495 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } attempt: 2
I20260620 14:35:40.568879 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.568769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46134)
0620 14:35:40.568820 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.568845 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:35:40.569190 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.569122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46136)
0620 14:35:40.569153 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.569171 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:35:40.569258 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40173: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40173: connect: Connection refused (error 111)
I20260620 14:35:40.582028 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:40.584376 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:40.584558 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:40.589479 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40173
I20260620 14:35:40.589525 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40173 every 8 connection(s)
I20260620 14:35:40.589869 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:35:40.591181 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:40.592692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22655
I20260620 14:35:40.592808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-1/wal/instance
I20260620 14:35:40.593022 392 external_mini_cluster.cc:1428] Setting key a479f1a55dd02e348e7bf6a827df5d50
I20260620 14:35:40.593222 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:35:40.602738 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.593958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.98.62:40239)
0620 14:35:40.594025 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.594029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.594269 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:35:40.594759 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.594761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.594775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.594855 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.595527 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.595994 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.597575 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.597772 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.597830 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.600122 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.600123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.600124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.600133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.601605 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.601951 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.601952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.601952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.602000 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.602226 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.602227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.602227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.602326 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.602376 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.602578 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.602644 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:35:40.603060 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.593936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.0.1:55590)
0620 14:35:40.594116 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.594165 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.594666 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.594904 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.594907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.595145 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.595438 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.595444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.596094 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.596095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.597492 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.597495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.597870 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.597872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.598043 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.598527 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.598536 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.600027 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.601675 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.601681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.601682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.601890 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.602057 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.602058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.602058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.602169 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.602380 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.602384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.602489 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.602567 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.602678 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:35:40.603607 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:40.611272 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.603912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.98.61:40173)
0620 14:35:40.604056 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.604061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.604079 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:35:40.604103 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.604109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.604119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.604181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.604487 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.605147 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.606092 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.606412 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.606513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.608394 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.608401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.608401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.608407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.610043 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.610429 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.610431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.610432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.610481 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.610686 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.610686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.610687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.610806 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.610865 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.611029 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.611144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 14:35:40.611276 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.603856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46138)
0620 14:35:40.603909 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.603922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.603964 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.604202 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.604203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.604360 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.604417 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.604418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.605224 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.605225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.606010 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.606012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.606507 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.606508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.606572 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.606990 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.606994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.608291 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.610127 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.610128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.610128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.610346 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.610552 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.610552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.610553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.610617 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.610873 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.610873 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.610924 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.610996 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.611099 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:35:40.611901 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:40.612257 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.612158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33720)
0620 14:35:40.612192 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.612240 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 14:35:40.612378 22672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
I20260620 14:35:40.612766 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.612688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33736)
0620 14:35:40.612723 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.612746 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 14:35:40.612910 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39371: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:35:40.619592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39371
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:40.660432 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } attempt: 3
I20260620 14:35:40.662122 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attempt: 1
I20260620 14:35:40.662528 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.662393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33742)
0620 14:35:40.662450 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.662491 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:35:40.662920 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.662810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33756)
0620 14:35:40.662859 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.662889 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:35:40.663030 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39371: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:40.667560 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.660828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46144)
0620 14:35:40.660879 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.660901 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.660974 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.661131 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.661132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.661298 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.661362 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.661364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.662059 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.662061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.662809 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.662811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.663313 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.663313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.663374 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.663809 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.663813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.665058 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.666718 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.666719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.666719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.666881 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.667067 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.667067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.667067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.667116 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.667343 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.667344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.667384 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.667433 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.667499 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:35:40.667610 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.660892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.0.98.61:40173)
0620 14:35:40.660963 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.660966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.660977 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:40.661036 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.661037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.661051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.661136 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.661470 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.661962 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.662909 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.663234 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.663307 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.665153 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.665155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.665155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.665162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.666637 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.666947 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.666948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.666949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.667005 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.667169 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.667170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.667170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.667288 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.667349 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.667457 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.667531 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:35:40.667948 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:40.668269 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.668173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33758)
0620 14:35:40.668219 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.668250 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:35:40.668566 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.668515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33760)
0620 14:35:40.668536 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.668554 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 14:35:40.668668 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39371: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
I20260620 14:35:40.699836 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attempt: 1
I20260620 14:35:40.700300 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.700155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33766)
0620 14:35:40.700219 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.700264 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:35:40.700660 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.700560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33772)
0620 14:35:40.700593 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.700625 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 14:35:40.700845 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39371: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
W20260620 14:35:40.701717 22730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:40.701896 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:40.701913 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:40.701972 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:40.703058 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:40.703096 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:40.703111 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:40.703125 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:40.703171 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:40.704658 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239,127.0.98.61:40173,127.0.98.60:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:40.704895 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:40.705122 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:40.709678 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:40.709698 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:40.709801 22730 server_base.cc:1034] running on GCE node
W20260620 14:35:40.709949 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:40.710177 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:40.710402 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:40.711578 22730 hybrid_clock.cc:648] HybridClock initialized: now 1781966140711565 us; error 26 us; skew 500 ppm
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:40.713932 22730 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:35:40.714373 22730 webserver.cc:460] Webserver started at http://127.0.98.60:42453/ using document root <none> and password file <none>
I20260620 14:35:40.714617 22730 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:40.714682 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:40.714824 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:40.715801 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data/instance:
uuid: "7702905bfb4a43d6ad173746bc7e2376"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "9fd2b69a05802c288393c3c7cd227b14"
server_key_iv: "c69bec5c4463b6fa2a83723fb89a260f"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.716138 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal/instance:
uuid: "7702905bfb4a43d6ad173746bc7e2376"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "9fd2b69a05802c288393c3c7cd227b14"
server_key_iv: "c69bec5c4463b6fa2a83723fb89a260f"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.717461 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:40.718206 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:40.718377 22730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:40.718469 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "7702905bfb4a43d6ad173746bc7e2376"
format_stamp: "Formatted at 2026-06-20 14:35:40 on dist-test-slave-tc2g"
server_key: "9fd2b69a05802c288393c3c7cd227b14"
server_key_iv: "c69bec5c4463b6fa2a83723fb89a260f"
server_key_version: "encryptionkey@0"
I20260620 14:35:40.718547 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:40.731163 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attempt: 2
I20260620 14:35:40.731525 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.731407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33776)
0620 14:35:40.731463 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.731500 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:35:40.731864 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.731774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33784)
0620 14:35:40.731818 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.731841 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:35:40.731945 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39371: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39371: connect: Connection refused (error 111)
I20260620 14:35:40.736033 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:40.737920 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:40.738075 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:40.742004 22730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39371
I20260620 14:35:40.742035 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39371 every 8 connection(s)
I20260620 14:35:40.742360 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:35:40.743222 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:40.743922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22730
I20260620 14:35:40.744040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965876904925-392-0/minicluster-data/master-2/wal/instance
I20260620 14:35:40.744239 392 external_mini_cluster.cc:1428] Setting key b5f89cb02faa0602a9b9e9ede708513e
I20260620 14:35:40.745110 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:35:40.754644 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.746099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.0.98.62:40239)
0620 14:35:40.746151 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.746153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.746302 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:35:40.746379 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.746380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.746391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.746445 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.746913 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.747607 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.748837 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.749112 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.749195 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.751609 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.751611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.751611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.751617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.753226 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.753699 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.753700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.753700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.753748 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.754011 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.754012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.754012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.754120 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.754195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.754478 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.754555 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:35:40.754814 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.746028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.0.1:55596)
0620 14:35:40.746182 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.746239 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.746318 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.746487 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.746490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.746740 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.746821 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.746826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.747724 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.747725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.748752 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.748754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.749215 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.749216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.749301 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.749888 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.749896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.751522 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.753309 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.753315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.753318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.753629 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.753814 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.753815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.753816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.753942 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.754199 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.754206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.754338 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.754440 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.754593 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260620 14:35:40.755388 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:40.755472 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.744742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.0.98.60:39371)
0620 14:35:40.744940 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.744949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.744980 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:35:40.745486 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.745496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.745558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.745706 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.746443 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.747650 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.748718 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.749188 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.749291 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.752217 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.752224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.752224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.752231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.754105 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.754425 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.754428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.754429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.754480 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.754765 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.754766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.754767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.754923 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.755024 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.755240 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.755311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:40.762096 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.755651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40173 (local address 127.0.0.1:46154)
0620 14:35:40.755699 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.755709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.755743 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.755876 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.755877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.756040 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.756082 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.756083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.756767 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.756767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.757364 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.757366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.757804 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.757805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.757868 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.758195 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.758200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.759456 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.761165 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.761165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.761166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.761301 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.761508 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.761508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.761509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.761558 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.761814 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.761815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.761853 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.761919 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.762003 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:35:40.762096 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.755724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.98.61:40173)
0620 14:35:40.755774 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.755776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.755785 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:40.755808 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.755809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.755818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.755873 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.756187 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.756686 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.757464 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.757736 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.757786 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.759538 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.759539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.759540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.759548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.761068 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.761375 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.761376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.761377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.761449 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.761640 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.761641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.761642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.761749 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.761822 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.761947 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.762015 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:35:40.762399 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:40.763595 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.756300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.98.62:40239)
0620 14:35:40.756335 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.756337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.756378 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:35:40.756445 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.756446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.756455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.756509 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.757029 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.757500 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.758305 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.758473 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.758523 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.760917 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.760918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.760918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.760925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.762506 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.762868 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.762869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.762869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.762979 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.763191 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.763193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.763193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.763291 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.763404 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.763489 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.763540 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:40.769345 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.762619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33796)
0620 14:35:40.762696 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.762708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.762770 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.762868 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.762868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.763027 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.763059 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.763061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.763896 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.763897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.764668 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.764670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.765059 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.765060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.765126 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.765528 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.765533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.766735 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.768295 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.768296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.768296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.768451 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.768661 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.768662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.768662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.768710 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.768984 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.768985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.769040 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.769107 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.769210 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:35:40.769443 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.762688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.0.98.60:39371)
0620 14:35:40.762717 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.762719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.762756 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:35:40.762808 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.762809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.762818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.762867 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.763129 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.763813 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.764764 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.764989 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.765062 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.766828 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.766829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.766829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.766835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.768203 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.768531 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.768532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.768533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.768597 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.768778 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.768779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.768780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.768911 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.768990 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.769157 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.769252 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":14}
I20260620 14:35:40.771142 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: Bootstrap starting.
I20260620 14:35:40.772094 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.764281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.98.61:40173)
0620 14:35:40.764313 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.764314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.764343 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:35:40.764363 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.764363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.764375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.764411 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.765024 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.765707 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.766409 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.766660 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.766740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.769035 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.769037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.769037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.769044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.771007 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.771302 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.771303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.771303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.771358 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.771552 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.771552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.771553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.771706 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.771890 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.771927 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.772004 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:35:40.772259 22798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:40.772553 22798 log.cc:824] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:40.773638 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: No bootstrap required, opened a new log
I20260620 14:35:40.773658 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:40.775137 22798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.775278 22798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7702905bfb4a43d6ad173746bc7e2376, State: Initialized, Role: FOLLOWER
I20260620 14:35:40.774662 22622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41aeef117bad4afdab5f6539dc2bd057"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55612
I20260620 14:35:40.775370 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.775725 22798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:40.775705 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.775776 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.776016 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attempt: 2
W20260620 14:35:40.777532 22815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:40.777837 22815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:35:40.778323 22730 master_runner.cc:186] Error getting master registration for 127.0.98.62:40239: OK, instance_id { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" instance_seqno: 1781966140431403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:40.781198 22730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:40.783733 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.776252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33798)
0620 14:35:40.776293 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.776306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.776379 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.776556 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.776557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.776758 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.776803 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.776805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.777664 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.777665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.778419 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.778422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.778849 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.778850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.778908 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.779318 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.779323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.780843 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.782658 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.782659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.782659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.782872 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.783082 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.783083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.783083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.783142 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.783435 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.783435 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.783476 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.783543 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.783636 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:35:40.783748 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.776280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.98.60:39371)
0620 14:35:40.776312 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.776314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.776365 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:35:40.776473 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.776474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.776485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.776552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.776880 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.777592 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.778485 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.778766 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.778836 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.780940 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.780942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.780942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.780948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.782566 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.782949 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.782950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.782951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.783007 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.783213 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.783214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.783214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.783365 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.783436 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.783585 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.783660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:35:40.785219 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: Bootstrap starting.
I20260620 14:35:40.785917 22648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:40.786201 22648 log.cc:824] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:40.786901 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: No bootstrap required, opened a new log
I20260620 14:35:40.787883 22648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.788014 22648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aeef117bad4afdab5f6539dc2bd057, State: Initialized, Role: FOLLOWER
I20260620 14:35:40.788105 22648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.788404 22648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:40.788544 22622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:35:40.788625 22622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:40.789463 22622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.789723 22622 leader_election.cc:288] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd2ec6bd01d4f4b8e3365d4fe5c25aa (127.0.98.61:40173), 7702905bfb4a43d6ad173746bc7e2376 (127.0.98.60:39371)
I20260620 14:35:40.790354 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.790457 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.790499 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41aeef117bad4afdab5f6539dc2bd057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7702905bfb4a43d6ad173746bc7e2376"
I20260620 14:35:40.790642 22772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:40.791586 22772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41aeef117bad4afdab5f6539dc2bd057 in term 1.
I20260620 14:35:40.791853 22597 leader_election.cc:302] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aeef117bad4afdab5f6539dc2bd057, 7702905bfb4a43d6ad173746bc7e2376; no voters:
I20260620 14:35:40.791960 22818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:40.792115 22818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [term 1 LEADER]: Becoming Leader. State: Replica: 41aeef117bad4afdab5f6539dc2bd057, State: Running, Role: LEADER
I20260620 14:35:40.792196 22818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.792582 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41aeef117bad4afdab5f6539dc2bd057. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.792665 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:40.791338 22580 master_runner.cc:186] Error getting master registration for 127.0.98.61:40173: OK, instance_id { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" instance_seqno: 1781966140588088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:40.793967 22580 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:35:40.790295 22829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:40.794095 22829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:40.794189 22837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:40.794435 22837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:40.796060 22772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [term 1 FOLLOWER]: Refusing update from remote peer 41aeef117bad4afdab5f6539dc2bd057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:40.796490 22835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:40.797770 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41aeef117bad4afdab5f6539dc2bd057. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.797870 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.806406 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.797019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:40239)
0620 14:35:40.797060 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.797062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.797872 (+ 810us) server_negotiation.cc:407] Connection header received
0620 14:35:40.797959 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.797960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.797972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.798036 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.798877 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.799341 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.800407 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.800723 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.800793 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.802495 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.802496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.802496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.802503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.804154 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.805165 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.805167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.805167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.805234 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.805605 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.805606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.805607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.805758 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.805854 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.806228 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.806314 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:40.810065 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.797451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.0.98.60:39371)
0620 14:35:40.797489 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.797491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.801555 (+ 4064us) server_negotiation.cc:407] Connection header received
0620 14:35:40.801657 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.801658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.801668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.801697 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.802347 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.803081 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.804193 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.804492 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.804555 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.806698 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.806699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.806711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.806718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.808343 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.808914 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.808915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.808915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.808975 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.809320 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.809321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.809321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.809480 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.809558 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.809882 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.809955 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:40.813127 22838 mvcc.cc:205] Tried to move back new op lower bound from 7298933312697417728 to 7298933312685522944. Current Snapshot: MvccSnapshot[applied={T|T < 7298933312697417728}]
I20260620 14:35:40.813509 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.797675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.98.61:40173)
0620 14:35:40.797718 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.797720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.805563 (+ 7843us) server_negotiation.cc:407] Connection header received
0620 14:35:40.805676 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.805677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.805687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.805717 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.806326 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.807044 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.808172 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.808507 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.808566 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.810060 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.810062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.810062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.810069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.812044 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.812450 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.812452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.812452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.812509 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.812712 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.812712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.812713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.812834 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.812955 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.813324 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.813401 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:40.814239 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.814323 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:40.814531 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.814603 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.814872 22841 mvcc.cc:205] Tried to move back new op lower bound from 7298933312697417728 to 7298933312685522944. Current Snapshot: MvccSnapshot[applied={T|T < 7298933312697417728}]
I20260620 14:35:40.821803 22837 catalog_manager.cc:1347] Generated new cluster ID: c8d94b0158ca4338b41a72f32a55ebc2
I20260620 14:35:40.821854 22837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:40.823057 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } attempt: 3
Jun 20 14:35:40 dist-test-slave-tc2g krb5kdc[22567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:35:40.830791 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.823416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.0.98.60:39371)
0620 14:35:40.823459 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.823461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.823467 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:35:40.823483 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.823484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.823491 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.823540 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.823824 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.824573 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.825589 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.825871 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.825944 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.828019 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:40.828020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:40.828020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:40.828027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:40.829690 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.830034 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.830035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.830035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.830084 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:40.830272 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:40.830273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:40.830274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:40.830427 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:40.830485 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:40.830641 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:35:40.830708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:40.830793 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:40.823299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39371 (local address 127.0.0.1:33804)
0620 14:35:40.823349 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:40.823361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:40.823409 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:40.823559 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:40.823560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:40.823714 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:40.823750 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.823752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.824664 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.824665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.825480 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:40.825482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:40.825944 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:40.825946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:40.826030 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:40.826492 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:40.826497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:40.827943 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:40.829756 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.829757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.829758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.829957 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.830133 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:40.830134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:40.830135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:40.830197 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:40.830489 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:40.830490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:40.830529 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:35:40.830619 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:35:40.830708 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:35:40.832582 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: Bootstrap starting.
I20260620 14:35:40.833304 22723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:40.833638 22723 log.cc:824] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:40.834479 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: No bootstrap required, opened a new log
I20260620 14:35:40.836625 22723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.836769 22723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd2ec6bd01d4f4b8e3365d4fe5c25aa, State: Initialized, Role: FOLLOWER
I20260620 14:35:40.836849 22723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } }
I20260620 14:35:40.837193 22723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:40.837229 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41aeef117bad4afdab5f6539dc2bd057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa"
I20260620 14:35:40.837311 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:40.838253 22697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41aeef117bad4afdab5f6539dc2bd057 in term 1.
I20260620 14:35:40.838631 22696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [term 1 FOLLOWER]: Refusing update from remote peer 41aeef117bad4afdab5f6539dc2bd057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 14:35:40.839767 22859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:40.839869 22859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:40.837148 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.840878 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.840972 22845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:40.842204 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41aeef117bad4afdab5f6539dc2bd057. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.842286 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.842331 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.842381 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:40.843402 22861 mvcc.cc:205] Tried to move back new op lower bound from 7298933312697417728 to 7298933312685522944. Current Snapshot: MvccSnapshot[applied={T|T < 7298933312697417728}]
I20260620 14:35:40.845825 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41aeef117bad4afdab5f6539dc2bd057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aeef117bad4afdab5f6539dc2bd057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } peers { permanent_uuid: "6bd2ec6bd01d4f4b8e3365d4fe5c25aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40173 } } peers { permanent_uuid: "7702905bfb4a43d6ad173746bc7e2376" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39371 } } }
I20260620 14:35:40.845934 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:40.849498 22837 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:40.850232 22837 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:40.857518 22837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41aeef117bad4afdab5f6539dc2bd057: Generated new TSK 0
I20260620 14:35:40.857740 22837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:35:40.882542 22801 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.881127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.0.98.60:39371)
0620 14:35:40.881165 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.881167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.881866 (+ 699us) server_negotiation.cc:407] Connection header received
0620 14:35:40.881990 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.881991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.882002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.882032 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.882472 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33808: BlockingRecv error: recv got EOF from 127.0.0.1:33808 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:35:40.883349 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.880904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.61:40173)
0620 14:35:40.880957 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.880960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.882835 (+ 1875us) server_negotiation.cc:407] Connection header received
0620 14:35:40.882945 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.882946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.882957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.882986 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.883289 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46172: BlockingRecv error: recv got EOF from 127.0.0.1:46172 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:35:40.884096 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.881538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.98.62:40239)
0620 14:35:40.881577 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.881579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.883605 (+ 2026us) server_negotiation.cc:407] Connection header received
0620 14:35:40.883700 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.883701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.883713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.883742 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.884040 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55616: BlockingRecv error: recv got EOF from 127.0.0.1:55616 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:35:40.886437 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.885659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.0.98.61:40173)
0620 14:35:40.885691 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.885693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.885965 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:35:40.886067 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.886068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.886083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.886112 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.886394 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46184: BlockingRecv error: recv got EOF from 127.0.0.1:46184 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:35:40.889382 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.888662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.98.61:40173)
0620 14:35:40.888691 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.888692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.888900 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:35:40.889003 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.889004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.889013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.889040 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.889322 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46186: BlockingRecv error: recv got EOF from 127.0.0.1:46186 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:35:40.893122 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.892434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.0.98.61:40173)
0620 14:35:40.892464 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.892465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.892654 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:35:40.892739 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.892739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.892748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.892775 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.893082 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46194: BlockingRecv error: recv got EOF from 127.0.0.1:46194 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:35:40.898833 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.898166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.0.98.61:40173)
0620 14:35:40.898197 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.898199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.898390 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:35:40.898480 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.898480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.898489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.898516 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.898792 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46204: BlockingRecv error: recv got EOF from 127.0.0.1:46204 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:35:40.908730 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.908009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.0.98.61:40173)
0620 14:35:40.908042 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.908044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.908247 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:35:40.908340 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.908341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.908352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.908381 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.908683 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46208: BlockingRecv error: recv got EOF from 127.0.0.1:46208 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:35:40.926875 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.926058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.98.61:40173)
0620 14:35:40.926095 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.926097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.926328 (+ 231us) server_negotiation.cc:407] Connection header received
0620 14:35:40.926426 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.926428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.926439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.926469 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.926821 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46222: BlockingRecv error: recv got EOF from 127.0.0.1:46222 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:35:40.961323 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:40.960342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.61:40173)
0620 14:35:40.960387 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:40.960389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:40.960656 (+ 267us) server_negotiation.cc:407] Connection header received
0620 14:35:40.960766 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:40.960767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:40.960779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:40.960809 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:40.961243 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46224: BlockingRecv error: recv got EOF from 127.0.0.1:46224 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:35:41.027705 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:41.026903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.0.98.61:40173)
0620 14:35:41.026995 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.026997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.027024 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:35:41.027102 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.027104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.027114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.027169 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:41.027605 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46226: BlockingRecv error: recv got EOF from 127.0.0.1:46226 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:35:41.157480 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:41.156776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.0.98.61:40173)
0620 14:35:41.156862 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.156864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.156925 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:35:41.156984 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.156985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.156996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.157038 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:41.157387 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46228: BlockingRecv error: recv got EOF from 127.0.0.1:46228 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:35:41.415264 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:41.414620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.98.61:40173)
0620 14:35:41.414685 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.414687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.414727 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:35:41.414787 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.414787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.414797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.414850 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:41.415178 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46242: BlockingRecv error: recv got EOF from 127.0.0.1:46242 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:35:41.778271 22815 catalog_manager.cc:1259] Loaded cluster ID: c8d94b0158ca4338b41a72f32a55ebc2
I20260620 14:35:41.778348 22815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: loading cluster ID for follower catalog manager: success
I20260620 14:35:41.779150 22815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: acquiring CA information for follower catalog manager: success
I20260620 14:35:41.779477 22815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7702905bfb4a43d6ad173746bc7e2376: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:41.840377 22859 catalog_manager.cc:1259] Loaded cluster ID: c8d94b0158ca4338b41a72f32a55ebc2
I20260620 14:35:41.840451 22859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: loading cluster ID for follower catalog manager: success
I20260620 14:35:41.841301 22859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: acquiring CA information for follower catalog manager: success
I20260620 14:35:41.841636 22859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bd2ec6bd01d4f4b8e3365d4fe5c25aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:41.931037 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:41.928348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.0.98.61:40173)
0620 14:35:41.928449 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.928451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.928464 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:35:41.928567 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.928568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.928637 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.928727 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:41.928887 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.929391 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.930217 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.930416 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.930501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:41.930674 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:41.930787 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:41.930868 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:41.930907 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:35:41.930926 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":41,"threads_started":1}
I20260620 14:35:41.935245 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:41.932991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.98.60:39371)
0620 14:35:41.933108 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.933110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.933118 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:35:41.933151 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.933152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.933182 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.933251 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:41.933331 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.933845 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.934529 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.934720 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.934772 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:41.934893 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:41.934997 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:41.935054 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:41.935100 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:35:41.935117 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260620 14:35:41.939630 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:41.936815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.60:39371)
0620 14:35:41.936853 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.936855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.937022 (+ 167us) server_negotiation.cc:407] Connection header received
0620 14:35:41.937081 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.937082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.937092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.937144 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:41.937237 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.937807 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.938882 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.939166 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.939225 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:41.939360 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:41.939386 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:41.939444 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:41.939527 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:35:41.939546 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:35:41.939939 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:41.936759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.0.98.61:40173)
0620 14:35:41.936887 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.936889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.936988 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:35:41.937078 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.937079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.937090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.937120 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:41.937292 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.938056 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.939026 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.939292 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.939346 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:41.939510 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:41.939540 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:41.939589 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:41.939844 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:35:41.939867 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 14:35:41.940095 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:41.936810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.0.98.62:40239)
0620 14:35:41.936931 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:41.936933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:41.937088 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:35:41.937136 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:41.937137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:41.937187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:41.937258 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:41.937478 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.938218 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.939122 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:41.939411 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:41.939465 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:41.939682 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:41.939784 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:41.939860 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:41.939904 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:35:41.939933 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260620 14:35:41.941550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22580
I20260620 14:35:41.946818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22655
I20260620 14:35:41.951503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22730
2026-06-20T14:35:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2782 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:42 dist-test-slave-tc2g krb5kdc[22946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:42 dist-test-slave-tc2g krb5kdc[22946](info): set up 2 sockets
Jun 20 14:35:42 dist-test-slave-tc2g krb5kdc[22946](info): commencing operation
krb5kdc: starting...
W20260620 14:35:42.993669 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/test-user.keytab.
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:35:43.106510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:43.185662 22959 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:43.186036 22959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:43.186074 22959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:43.186118 22959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:43.187244 22959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:43.187310 22959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:43.187333 22959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:43.187355 22959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:43.187382 22959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:43.188841 22959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:43.189116 22959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:43.189332 22959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:43.194067 22965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:43.194093 22967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:43.194216 22959 server_base.cc:1034] running on GCE node
W20260620 14:35:43.194064 22964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:43.194597 22959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:43.194882 22959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:43.196055 22959 hybrid_clock.cc:648] HybridClock initialized: now 1781966143196026 us; error 38 us; skew 500 ppm
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:43.198514 22959 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:35:43.198870 22959 webserver.cc:460] Webserver started at http://127.0.98.62:42919/ using document root <none> and password file <none>
I20260620 14:35:43.199034 22959 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:43.199083 22959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:43.199167 22959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:43.200124 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data/instance:
uuid: "bd9736c304c242d7b43936fcefc8ddd3"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "33708acefe6741318d4e758618f297c0"
server_key_iv: "09f86a274dc387284c83e16257c1b341"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.200426 22959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd9736c304c242d7b43936fcefc8ddd3"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "33708acefe6741318d4e758618f297c0"
server_key_iv: "09f86a274dc387284c83e16257c1b341"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.201660 22959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:35:43.202435 22974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:43.202656 22959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:35:43.202754 22959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal
uuid: "bd9736c304c242d7b43936fcefc8ddd3"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "33708acefe6741318d4e758618f297c0"
server_key_iv: "09f86a274dc387284c83e16257c1b341"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.202838 22959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:43.218532 22959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:43.220877 22959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:43.221038 22959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:43.225097 22959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33089
I20260620 14:35:43.225158 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33089 every 8 connection(s)
I20260620 14:35:43.225553 22959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:43.226523 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:43.228060 23027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:43.230902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 22959
I20260620 14:35:43.230980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-0/wal/instance
I20260620 14:35:43.231187 392 external_mini_cluster.cc:1428] Setting key 195aa0e4d44d6b1ba7645fac32d8bdea
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:43.238076 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.228781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.0.98.62:33089)
0620 14:35:43.228935 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.228943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.229043 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:35:43.229668 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.229670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.229692 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.229785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.230325 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.231224 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.232084 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.232392 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.232513 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.234939 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.234945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.234946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.234952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.236471 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.236926 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.236928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.236928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.236999 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.237256 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.237257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.237257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.237513 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.237579 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.237804 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.237900 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260620 14:35:43.238126 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.228717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33089 (local address 127.0.0.1:57454)
0620 14:35:43.228926 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.228969 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.229498 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.229790 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.229793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.230075 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.230220 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.230224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.231312 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.231312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.231987 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.231989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.232499 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.232500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.232567 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.233107 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.233115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.234856 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.236557 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.236563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.236565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.236840 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.237066 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.237067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.237068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.237188 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.237581 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.237588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.237671 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.237779 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.237972 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260620 14:35:43.238860 23027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 14:35:43.239244 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.239102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37398)
0620 14:35:43.239159 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.239212 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:35:43.239382 22975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
I20260620 14:35:43.239802 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.239719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37404)
0620 14:35:43.239761 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.239786 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:35:43.239919 23027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:35:43.254384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34733
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:43.289126 23027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } attempt: 1
I20260620 14:35:43.289736 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.289556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37416)
0620 14:35:43.289633 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.289697 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:35:43.290122 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.289994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37420)
0620 14:35:43.290070 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.290098 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:35:43.290223 23027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
W20260620 14:35:43.333449 23034 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:43.333652 23034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:43.333669 23034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:43.333688 23034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:43.334829 23034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:43.334872 23034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:43.334887 23034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:43.334928 23034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:43.334959 23034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:43.336402 23034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:43.336652 23034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:43.336874 23034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:43.341619 23034 server_base.cc:1034] running on GCE node
W20260620 14:35:43.341630 23042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:43.341572 23040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:43.341571 23039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:43.342051 23034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:43.342370 23034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:43.343529 23034 hybrid_clock.cc:648] HybridClock initialized: now 1781966143343521 us; error 40 us; skew 500 ppm
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:43.345834 23034 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:35:43.346257 23034 webserver.cc:460] Webserver started at http://127.0.98.61:41135/ using document root <none> and password file <none>
I20260620 14:35:43.346482 23034 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:43.346544 23034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:43.346676 23034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:43.347713 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data/instance:
uuid: "c2659774ce03481e85d71a278d207712"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "97601992b81e0ceb0a05870aa521770a"
server_key_iv: "7c214c878a8b6fe167d77d556139b5de"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.348109 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal/instance:
uuid: "c2659774ce03481e85d71a278d207712"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "97601992b81e0ceb0a05870aa521770a"
server_key_iv: "7c214c878a8b6fe167d77d556139b5de"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.349346 23034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:35:43.350049 23049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:43.350220 23034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:43.350311 23034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal
uuid: "c2659774ce03481e85d71a278d207712"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "97601992b81e0ceb0a05870aa521770a"
server_key_iv: "7c214c878a8b6fe167d77d556139b5de"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.350404 23034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:43.358355 23027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } attempt: 2
I20260620 14:35:43.358758 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.358635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37428)
0620 14:35:43.358691 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.358728 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:35:43.359076 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.358998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37434)
0620 14:35:43.359036 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.359060 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:35:43.359177 23027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34733: connect: Connection refused (error 111)
I20260620 14:35:43.368930 23034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:43.370801 23034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:43.370945 23034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:43.374709 23034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34733
I20260620 14:35:43.374795 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34733 every 8 connection(s)
I20260620 14:35:43.375082 23034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:35:43.376037 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:43.377578 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:43.378546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 23034
I20260620 14:35:43.378643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-1/wal/instance
I20260620 14:35:43.378845 392 external_mini_cluster.cc:1428] Setting key bd4a33b8923426c1202fad208f0b5d20
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:35:43.387269 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.378245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.98.62:33089)
0620 14:35:43.378305 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.378308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.378576 (+ 268us) server_negotiation.cc:407] Connection header received
0620 14:35:43.379282 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.379283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.379295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.379333 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.380199 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.380948 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.381983 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.382343 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.382424 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.384401 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.384403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.384403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.384411 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.385997 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.386388 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.386389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.386390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.386453 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.386728 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.386729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.386730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.386853 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.386909 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.387142 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.387206 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:35:43.387569 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.378227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33089 (local address 127.0.0.1:57466)
0620 14:35:43.378427 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.378476 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.379170 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.379418 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.379422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.379812 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.380082 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.380088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.381058 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.381059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.381898 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.381900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.382436 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.382437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.382546 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.383026 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.383033 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.384313 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.386105 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.386112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.386113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.386314 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.386533 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.386534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.386535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.386656 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.386915 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.386920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.387031 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.387119 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.387233 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260620 14:35:43.388114 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:43.395599 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.388401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.98.61:34733)
0620 14:35:43.388561 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.388565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.388579 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:35:43.388597 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.388598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.388608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.388701 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.388978 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.389706 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.390362 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.390759 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.390851 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.392791 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.392798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.392798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.392804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.394399 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.394782 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.394784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.394784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.394833 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.395022 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.395023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.395023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.395144 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.395204 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.395376 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.395497 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260620 14:35:43.395648 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.388348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37448)
0620 14:35:43.388393 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.388404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.388442 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.388701 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.388702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.388843 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.388906 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.388908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.389793 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.389793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.390275 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.390276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.390850 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.390851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.390917 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.391311 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.391316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.392710 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.394484 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.394485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.394486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.394697 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.394896 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.394897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.394898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.394962 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.395214 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.395215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.395266 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.395346 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.395453 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:35:43.396358 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:43.396741 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.396609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46334)
0620 14:35:43.396687 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.396724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 14:35:43.396874 23053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
I20260620 14:35:43.397225 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.397133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46348)
0620 14:35:43.397180 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.397202 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:35:43.397325 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:35:43.403735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpSwtz3/build/release/bin/kudu
/tmp/dist-test-taskpSwtz3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39577
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:43.446583 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } attempt: 1
I20260620 14:35:43.447084 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.446912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46364)
0620 14:35:43.446992 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.447039 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:35:43.447472 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.447380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46378)
0620 14:35:43.447425 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.447452 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:35:43.447595 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
I20260620 14:35:43.450320 23027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } attempt: 3
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:43.457604 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.450576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37458)
0620 14:35:43.450628 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.450639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.450711 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.450856 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.450857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.451017 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.451080 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.451082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.451766 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.451767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.452284 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.452286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.452795 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.452796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.452880 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.453382 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.453387 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.454885 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.456508 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.456509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.456510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.456683 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.456895 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.456896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.456897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.456960 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.457240 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.457241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.457290 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.457379 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.457507 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:35:43.457602 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.450651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.61:34733)
0620 14:35:43.450711 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.450713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.450724 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:43.450774 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.450774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.450784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.450852 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.451181 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.451677 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.452380 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.452721 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.452796 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.454980 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.454981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.454981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.454988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.456401 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.456771 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.456772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.456773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.456828 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.457026 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.457027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.457027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.457162 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.457248 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.457402 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.457510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:35:43.457980 23027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:43.458479 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.458370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46390)
0620 14:35:43.458422 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.458456 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:35:43.458871 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.458781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46406)
0620 14:35:43.458825 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.458849 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:35:43.459002 23027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
W20260620 14:35:43.487126 23109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:43.487345 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:43.487375 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:43.487416 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:43.488498 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:43.488569 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:43.488597 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:43.488628 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:43.488651 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:43.490139 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33089,127.0.98.61:34733,127.0.98.60:39577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:12:33 UTC on ccycloud.quasar-ozjbme.root.comops.site
build id 4457461
I20260620 14:35:43.490132 23027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } attempt: 1
I20260620 14:35:43.490392 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:43.490531 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.490407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46414)
0620 14:35:43.490470 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.490500 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:35:43.490648 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:43.490908 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.490805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46424)
0620 14:35:43.490855 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.490884 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:35:43.490996 23027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
W20260620 14:35:43.495437 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:43.495414 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:43.495414 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:43.495693 23109 server_base.cc:1034] running on GCE node
I20260620 14:35:43.495929 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:43.496193 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:43.497329 23109 hybrid_clock.cc:648] HybridClock initialized: now 1781966143497311 us; error 41 us; skew 500 ppm
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:43.499898 23109 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:35:43.500389 23109 webserver.cc:460] Webserver started at http://127.0.98.60:45795/ using document root <none> and password file <none>
I20260620 14:35:43.500595 23109 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:43.500639 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:43.500731 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:43.501847 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data/instance:
uuid: "17789ce0870845fb996495722d965e79"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "eee579b9be024d28ad60006662c1c652"
server_key_iv: "9ea5761b8fad8464e1dd248f61ba4027"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.502354 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal/instance:
uuid: "17789ce0870845fb996495722d965e79"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "eee579b9be024d28ad60006662c1c652"
server_key_iv: "9ea5761b8fad8464e1dd248f61ba4027"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.503684 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:35:43.504364 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:43.504539 23109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:35:43.504623 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal
uuid: "17789ce0870845fb996495722d965e79"
format_stamp: "Formatted at 2026-06-20 14:35:43 on dist-test-slave-tc2g"
server_key: "eee579b9be024d28ad60006662c1c652"
server_key_iv: "9ea5761b8fad8464e1dd248f61ba4027"
server_key_version: "encryptionkey@0"
I20260620 14:35:43.504696 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:43.515764 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } attempt: 2
I20260620 14:35:43.516146 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.516016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46430)
0620 14:35:43.516074 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.516113 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:35:43.516484 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.516407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46434)
0620 14:35:43.516446 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.516468 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:35:43.516597 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39577: connect: Connection refused (error 111)
I20260620 14:35:43.531013 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:43.532799 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:43.532998 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:43.536976 23109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39577
I20260620 14:35:43.537031 23176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39577 every 8 connection(s)
I20260620 14:35:43.537360 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:35:43.538236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpSwtz3/build/release/bin/kudu as pid 23109
I20260620 14:35:43.538336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpSwtz3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965876904925-392-0/minicluster-data/master-2/wal/instance
I20260620 14:35:43.538482 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:43.538563 392 external_mini_cluster.cc:1428] Setting key c4cf539394286702874a2a4c48ebec78
I20260620 14:35:43.540764 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:35:43.549103 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.541477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.98.62:33089)
0620 14:35:43.541519 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.541521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.541680 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:35:43.541773 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.541774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.541784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.541841 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.542269 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.542730 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.543478 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.543748 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.543813 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.545900 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.545902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.545902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.545910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.547631 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.548169 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.548170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.548171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.548218 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.548484 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.548485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.548485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.548599 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.548677 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.548938 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.549012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:35:43.549283 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.541406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33089 (local address 127.0.0.1:57478)
0620 14:35:43.541573 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.541618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.541708 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.541859 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.541862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.542109 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.542177 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.542179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.542816 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.542817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.543390 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.543392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.543850 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.543852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.543933 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.544323 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.544334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.545815 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.547721 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.547725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.547727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.548095 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.548296 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.548297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.548298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.548413 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.548668 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.548674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.548811 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.548916 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.549060 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260620 14:35:43.549603 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.539098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.0.98.60:39577)
0620 14:35:43.539280 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.539289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.539322 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:35:43.539906 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.539913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.539958 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.540077 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.540873 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.542029 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.542791 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.543195 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.543305 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.546457 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.546463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.546464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.546470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.548157 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.548483 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.548486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.548486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.548551 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.548749 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.548750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.548750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.548885 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.549027 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.549113 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.549294 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260620 14:35:43.550055 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:35:43.556844 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.550297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34733 (local address 127.0.0.1:37470)
0620 14:35:43.550361 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.550371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.550407 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.550551 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.550551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.550653 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.550679 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.550680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.551305 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.551306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.551775 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.551776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.552156 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.552156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.552213 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.552646 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.552650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.553982 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.555820 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.555821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.555821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.555975 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.556181 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.556182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.556182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.556230 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.556481 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.556481 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.556544 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.556640 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.556727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:35:43.556843 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.550397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.0.98.61:34733)
0620 14:35:43.550442 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.550444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.550455 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:43.550479 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.550480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.550491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.550546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.550757 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.551215 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.551855 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.552075 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.552156 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.554080 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.554081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.554082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.554088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.555737 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.556054 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.556055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.556056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.556113 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.556287 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.556288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.556288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.556419 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.556487 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.556659 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.556734 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:35:43.557211 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:43.557817 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.550386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.98.62:33089)
0620 14:35:43.550582 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.550584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.550590 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:35:43.550617 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.550617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.550624 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.550685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.551123 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.551548 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.552226 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.552527 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.552591 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.555009 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.555010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.555010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.555018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.556454 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.556804 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.556805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.556805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.556856 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.557075 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.557076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.557076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.557198 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.557349 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.557555 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.557633 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:35:43.563733 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.557470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46456)
0620 14:35:43.557543 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.557553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.557593 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.557706 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.557707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.557824 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.557848 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.557850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.558697 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.558698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.559166 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.559167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.559545 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.559545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.559592 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.559966 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.559970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.561005 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.562659 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.562660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.562660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.562871 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.563072 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.563073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.563073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.563130 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.563376 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.563377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.563422 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.563498 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.563568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:35:43.563825 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.557502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.98.60:39577)
0620 14:35:43.557538 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.557540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.557581 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:35:43.557637 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.557637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.557646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.557704 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.557931 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.558625 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.559253 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.559465 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.559553 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.561107 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.561108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.561108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.561114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.562578 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.562943 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.562944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.562945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.563002 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.563193 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.563194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.563194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.563312 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.563359 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.563533 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.563635 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":18}
I20260620 14:35:43.565306 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: Bootstrap starting.
I20260620 14:35:43.566061 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.558481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.0.98.61:34733)
0620 14:35:43.558531 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.558533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.558542 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:35:43.558601 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.558602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.558617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.558668 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.559100 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.559541 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.560350 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.560561 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.560650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.563224 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.563225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.563226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.563232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.564845 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.565213 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.565214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.565214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.565259 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.565582 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.565583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.565583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.565690 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.565803 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.565950 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.566002 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:35:43.566136 23027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } attempt: 2
I20260620 14:35:43.566411 23177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:43.566720 23177 log.cc:824] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:43.567540 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: No bootstrap required, opened a new log
I20260620 14:35:43.567586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:43.569413 23177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.569594 23177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17789ce0870845fb996495722d965e79, State: Initialized, Role: FOLLOWER
I20260620 14:35:43.569700 23177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.570066 23177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:43.569164 23001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bd9736c304c242d7b43936fcefc8ddd3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57490
I20260620 14:35:43.570331 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.570415 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260620 14:35:43.572005 23194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:43.572057 23194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:43.573993 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.566367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46462)
0620 14:35:43.566412 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.566424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.566465 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.567086 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.567087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.567245 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.567289 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.567291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.568090 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.568092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.568633 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.568635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.569077 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.569079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.569142 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.569580 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.569586 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.571002 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.572832 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.572833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.572834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.573083 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.573282 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.573283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.573284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.573338 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.573723 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.573724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.573769 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.573815 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.573902 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:35:43.574295 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.566402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.98.60:39577)
0620 14:35:43.566977 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.566979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.566987 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:35:43.567005 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.567006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.567014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.567077 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.567369 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.567992 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.568715 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.568991 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.569070 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.571087 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.571088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.571089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.571096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.572739 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.573166 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.573167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.573167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.573215 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.573403 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.573404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.573404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.573606 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.574112 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.574153 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.574220 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260620 14:35:43.572223 23109 master_runner.cc:186] Error getting master registration for 127.0.98.62:33089: OK, instance_id { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" instance_seqno: 1781966143223964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:43.574971 23109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:43.575703 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3: Bootstrap starting.
I20260620 14:35:43.576401 23027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:43.576691 23027 log.cc:824] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:43.577455 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3: No bootstrap required, opened a new log
I20260620 14:35:43.578735 23027 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.578889 23027 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9736c304c242d7b43936fcefc8ddd3, State: Initialized, Role: FOLLOWER
I20260620 14:35:43.578970 23027 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.579269 23027 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:43.579324 23001 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:35:43.579254 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.579383 23001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:43.580132 23001 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.580356 23001 leader_election.cc:288] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [CANDIDATE]: Term 1 election: Requested vote from peers c2659774ce03481e85d71a278d207712 (127.0.98.61:34733), 17789ce0870845fb996495722d965e79 (127.0.98.60:39577)
I20260620 14:35:43.580607 23151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd9736c304c242d7b43936fcefc8ddd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "17789ce0870845fb996495722d965e79"
I20260620 14:35:43.580705 23151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:43.581451 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.581578 23151 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd9736c304c242d7b43936fcefc8ddd3 in term 1.
I20260620 14:35:43.581846 22978 leader_election.cc:302] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17789ce0870845fb996495722d965e79, bd9736c304c242d7b43936fcefc8ddd3; no voters:
I20260620 14:35:43.582003 23197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:43.582301 23197 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [term 1 LEADER]: Becoming Leader. State: Replica: bd9736c304c242d7b43936fcefc8ddd3, State: Running, Role: LEADER
I20260620 14:35:43.582388 23197 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.582809 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9736c304c242d7b43936fcefc8ddd3. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.582880 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:43.583643 23216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:43.583858 22959 master_runner.cc:186] Error getting master registration for 127.0.98.61:34733: OK, instance_id { permanent_uuid: "c2659774ce03481e85d71a278d207712" instance_seqno: 1781966143373715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:43.586833 22959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:43.587379 23216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:43.589612 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.581722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.0.98.61:34733)
0620 14:35:43.581822 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.581824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.581869 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:35:43.582145 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.582146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.582157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.582205 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.583019 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.583705 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.584766 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.585061 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.585102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.586709 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.586710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.586710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.586717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.588403 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.588718 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.588720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.588720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.588773 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.589021 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.589022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.589022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.589169 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.589259 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.589409 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.589503 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:35:43.598542 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.589643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.62:33089)
0620 14:35:43.589681 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.589683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.590120 (+ 437us) server_negotiation.cc:407] Connection header received
0620 14:35:43.590226 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.590227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.590240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.590294 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.591072 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.591766 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.592681 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.592984 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.593023 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.595078 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.595079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.595080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.595088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.596756 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.597384 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.597386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.597386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.597478 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.597794 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.597795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.597796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.597937 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.598011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.598355 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.598445 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:43.598932 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.589910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.98.60:39577)
0620 14:35:43.589950 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.589952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.590296 (+ 344us) server_negotiation.cc:407] Connection header received
0620 14:35:43.590358 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.590359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.590384 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.590434 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.591407 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.592174 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.593765 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.594070 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.594114 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.596085 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.596086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.596086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.596093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.597750 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.598109 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.598109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.598110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.598162 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.598366 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.598366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.598367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.598485 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.598546 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.598779 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.598848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:35:43.600281 23151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [term 1 FOLLOWER]: Refusing update from remote peer bd9736c304c242d7b43936fcefc8ddd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:43.600651 23214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:43.602037 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9736c304c242d7b43936fcefc8ddd3. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.602131 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.606393 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.606489 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:43.606487 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.606554 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.606575 23218 mvcc.cc:205] Tried to move back new op lower bound from 7298933324184145920 to 7298933324114116608. Current Snapshot: MvccSnapshot[applied={T|T < 7298933324184145920}]
I20260620 14:35:43.606791 23220 mvcc.cc:205] Tried to move back new op lower bound from 7298933324184145920 to 7298933324114116608. Current Snapshot: MvccSnapshot[applied={T|T < 7298933324184145920}]
I20260620 14:35:43.607730 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } attempt: 3
Jun 20 14:35:43 dist-test-slave-tc2g krb5kdc[22946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:35:43.614146 23216 catalog_manager.cc:1347] Generated new cluster ID: a4ba32a8c5b14501a5dad301d2f75f09
I20260620 14:35:43.614192 23216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:43.618582 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.607963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39577 (local address 127.0.0.1:46478)
0620 14:35:43.608004 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:43.608014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:43.608049 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:43.608288 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:43.608289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:43.608453 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:43.608496 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.608498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.609329 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.609330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.610149 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:43.610150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.610610 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.610611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:43.610682 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.611163 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:43.611198 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:43.612730 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:43.617530 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.617531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.617531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.617758 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.617969 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:43.617970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:43.617970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:43.618033 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:43.618302 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:43.618303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:43.618354 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:35:43.618411 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:35:43.618493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:35:43.619055 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:43.608146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.98.60:39577)
0620 14:35:43.608178 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.608179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.608186 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:35:43.608222 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.608223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.608231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.608279 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.608569 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.609260 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.610244 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:43.610536 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:43.610592 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:43.612813 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:43.612814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:43.612814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:43.612820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:43.614441 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.617829 (+ 3388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.617829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.617830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.617891 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:43.618098 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:43.618098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:43.618099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:43.618222 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:43.618267 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:43.618900 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 14:35:43.618970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:35:43.620414 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: Bootstrap starting.
I20260620 14:35:43.621171 23102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:43.621515 23102 log.cc:824] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:43.622360 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: No bootstrap required, opened a new log
I20260620 14:35:43.624136 23102 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.624259 23102 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2659774ce03481e85d71a278d207712, State: Initialized, Role: FOLLOWER
I20260620 14:35:43.624336 23102 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } }
I20260620 14:35:43.624611 23102 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:43.625507 23075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:43.626281 23075 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [term 1 FOLLOWER]: Refusing update from remote peer bd9736c304c242d7b43936fcefc8ddd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:43.626447 23076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd9736c304c242d7b43936fcefc8ddd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2659774ce03481e85d71a278d207712"
I20260620 14:35:43.626575 23223 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:43.627234 23076 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd9736c304c242d7b43936fcefc8ddd3 in term 1.
W20260620 14:35:43.629514 23241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:43.629563 23241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:43.629690 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.629752 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.629798 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9736c304c242d7b43936fcefc8ddd3. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.629848 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.629881 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.629928 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:43.632484 23233 mvcc.cc:205] Tried to move back new op lower bound from 7298933324184145920 to 7298933324114116608. Current Snapshot: MvccSnapshot[applied={T|T < 7298933324184145920}]
I20260620 14:35:43.633481 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd9736c304c242d7b43936fcefc8ddd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd9736c304c242d7b43936fcefc8ddd3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } peers { permanent_uuid: "c2659774ce03481e85d71a278d207712" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34733 } } peers { permanent_uuid: "17789ce0870845fb996495722d965e79" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39577 } } }
I20260620 14:35:43.633565 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:43.634652 23216 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:43.635332 23216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:43.642915 23216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9736c304c242d7b43936fcefc8ddd3: Generated new TSK 0
I20260620 14:35:43.643090 23216 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:35:43.668270 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.666572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.98.60:39577)
0620 14:35:43.666618 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.666621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.667610 (+ 989us) server_negotiation.cc:407] Connection header received
0620 14:35:43.667718 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.667720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.667731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.667761 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.668203 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46480: BlockingRecv error: recv got EOF from 127.0.0.1:46480 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:35:43.669009 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.666998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.98.62:33089)
0620 14:35:43.667037 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.667039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.668502 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 14:35:43.668604 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.668606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.668616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.668647 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.668950 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57514: BlockingRecv error: recv got EOF from 127.0.0.1:57514 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:35:43.669836 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.667335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.0.98.61:34733)
0620 14:35:43.667372 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.667374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.669238 (+ 1864us) server_negotiation.cc:407] Connection header received
0620 14:35:43.669338 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.669339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.669350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.669380 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.669758 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37506: BlockingRecv error: recv got EOF from 127.0.0.1:37506 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:35:43.672358 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.671489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.98.61:34733)
0620 14:35:43.671523 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.671525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.671763 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:35:43.671948 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.671949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.671967 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.671999 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.672316 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37508: BlockingRecv error: recv got EOF from 127.0.0.1:37508 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:35:43.675165 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.674480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.0.98.61:34733)
0620 14:35:43.674510 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.674512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.674702 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:35:43.674792 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.674793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.674802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.674830 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.675125 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37510: BlockingRecv error: recv got EOF from 127.0.0.1:37510 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:35:43.678931 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.678233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.0.98.61:34733)
0620 14:35:43.678266 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.678268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.678466 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:35:43.678561 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.678562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.678572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.678600 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.678892 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37520: BlockingRecv error: recv got EOF from 127.0.0.1:37520 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:35:43.684820 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.684030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.0.98.61:34733)
0620 14:35:43.684076 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.684078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.684296 (+ 218us) server_negotiation.cc:407] Connection header received
0620 14:35:43.684420 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.684421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.684431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.684459 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.684780 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37522: BlockingRecv error: recv got EOF from 127.0.0.1:37522 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:35:43.694701 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.693957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.61:34733)
0620 14:35:43.693992 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.693993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.694212 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:35:43.694307 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.694308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.694319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.694346 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.694658 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37538: BlockingRecv error: recv got EOF from 127.0.0.1:37538 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:35:43.712805 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.711944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.61:34733)
0620 14:35:43.711984 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.711987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.712237 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:35:43.712331 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.712332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.712343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.712370 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.712745 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37540: BlockingRecv error: recv got EOF from 127.0.0.1:37540 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:35:43.746893 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.746160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.0.98.61:34733)
0620 14:35:43.746207 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.746209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.746327 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:35:43.746413 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.746414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.746426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.746479 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.746821 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37552: BlockingRecv error: recv got EOF from 127.0.0.1:37552 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:35:43.812649 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.811946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.0.98.61:34733)
0620 14:35:43.812002 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.812004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.812127 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:35:43.812190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.812191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.812201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.812265 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.812568 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37560: BlockingRecv error: recv got EOF from 127.0.0.1:37560 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:35:43.942739 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:43.941937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.98.61:34733)
0620 14:35:43.942023 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:43.942026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:43.942129 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:35:43.942216 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:43.942217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:43.942230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:43.942304 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:43.942669 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37566: BlockingRecv error: recv got EOF from 127.0.0.1:37566 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 14:35:44.200650 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:44.199919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.0.98.61:34733)
0620 14:35:44.199969 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.199971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.200062 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:35:44.200140 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.200141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.200162 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.200235 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:44.200566 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37576: BlockingRecv error: recv got EOF from 127.0.0.1:37576 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:35:44.572463 23194 catalog_manager.cc:1259] Loaded cluster ID: a4ba32a8c5b14501a5dad301d2f75f09
I20260620 14:35:44.572547 23194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: loading cluster ID for follower catalog manager: success
I20260620 14:35:44.573385 23194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: acquiring CA information for follower catalog manager: success
I20260620 14:35:44.573763 23194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17789ce0870845fb996495722d965e79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:44.629973 23241 catalog_manager.cc:1259] Loaded cluster ID: a4ba32a8c5b14501a5dad301d2f75f09
I20260620 14:35:44.630045 23241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: loading cluster ID for follower catalog manager: success
I20260620 14:35:44.630889 23241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: acquiring CA information for follower catalog manager: success
I20260620 14:35:44.631210 23241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2659774ce03481e85d71a278d207712: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:44.716262 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:44.713875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.0.98.61:34733)
0620 14:35:44.713971 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.713972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.713982 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:44.714028 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.714029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.714088 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.714171 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:44.714322 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.714822 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.715561 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.715747 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.715794 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:44.715936 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:44.716042 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:44.716097 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:44.716136 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:35:44.716151 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
I20260620 14:35:44.720237 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:44.718116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.98.60:39577)
0620 14:35:44.718233 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.718235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.718242 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:35:44.718274 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.718275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.718307 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.718372 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:44.718451 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.718910 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.719550 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.719756 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.719804 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:44.719929 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:44.720016 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:44.720071 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:44.720108 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:35:44.720124 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260620 14:35:44.724661 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:44.721944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.0.98.61:34733)
0620 14:35:44.721989 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.721990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.722102 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:35:44.722149 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.722150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.722161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.722221 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:44.722317 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.723015 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.723849 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.724141 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.724203 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:44.724348 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:44.724405 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:44.724455 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:44.724544 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:35:44.724565 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:35:44.724933 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:44.721893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.98.60:39577)
0620 14:35:44.721938 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.721940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.722175 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:35:44.722208 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.722208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.722218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.722266 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:44.722415 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.722979 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.723983 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.724269 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.724342 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:44.724637 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:44.724670 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:44.724719 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:44.724826 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:35:44.724850 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:35:44.725453 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:44.721823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.98.62:33089)
0620 14:35:44.722259 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:44.722261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:44.722269 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:35:44.722309 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:44.722311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:44.722369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:44.722468 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:44.722565 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.723299 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.724251 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:44.724567 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:44.724686 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:44.725012 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:44.725189 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:44.725249 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:44.725293 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:35:44.725322 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":52,"threads_started":1}
I20260620 14:35:44.732487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 22959
I20260620 14:35:44.737851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 23034
I20260620 14:35:44.742586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpSwtz3/build/release/bin/kudu with pid 23109
2026-06-20T14:35:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2790 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5572 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267839 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST